-
Notifications
You must be signed in to change notification settings - Fork 3.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
allow a script to be a part of inline commands in shell provisioner #13313
base: main
Are you sure you want to change the base?
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hey @anurag5sh,
I don't know if adding an extra argument to this provisioner is the way to go here, as both content
and inline
are essentially equivalent, with the difference that inline
adds a shebang automatically, and that's what caused the problem reported in the related issue in the first place.
Rather than introducing one more argument to the provisioner's configuration, I would think we can check that a shebang is not already present in the inline
, and if one's there we shouldn't try to inject our own?
This would probably only require an additional check on InlineShebang
, so we leave it empty assuming there's a shebang in the provided inline script.
Although this check is done during Prepare
, so we could end-up in a situation where it's not detected at that time if there's some interpolation in the command, so this might be better served in the Provision
function, when we write the temporary script for the inline
commands.
What do you think?
I agree, this would work perfectly for most scenarios. But it also introduces some workflow where it can lead to failures due to incorrect usage. Scenario 1: If the user has multiple commands within inline, the templatefile script (or any script) must always be the first command. Otherwise it becomes little messy to determine the right shebang from the commands and place it on first line.
Scenario 2: If someone chains multiple scripts within inline, the subsequent shebang's may be ignored (ig only the first shebang on line 1 is considered). Though this is rare and should be fine if both shebangs are same.
We must ensure that the users understand, to execute the commands in another provisioner block in such cases. |
To respond to your scenarios: the "inline" statements are glued together as one final script, if you use |
Yeah, I think avoiding to inject a shebang if already present would work just fine. But in order to scan for the shebang in commands, I think we can check in all the commands provided.
In this inline, the 2nd cmd has the shebang which we need to take from. We can continue to check for the shebang until the first match is found What do u think? |
I wouldn't try to infer a shebang based on parts of the script, if a user decides to add stuff in front of it, I would assume this is done with an intent, and if they need to add their own shebang, they should do so through the |
So based on the above discussions, this is how the
|
bc91a1b
to
f574090
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hey @anurag5sh,
Overall this is better of an approach imho compared to adding a new option for this, but I think there are some changes to do still, mostly I would advise moving the shebang detection logic to the Provision
function instead of the Prepare
function, because there might be some interpolation context that gets populated after the VM gets created, which is not yet known while Prepare
is invoked.
provisioner/shell/provisioner.go
Outdated
@@ -112,7 +112,16 @@ func (p *Provisioner) Prepare(raws ...interface{}) error { | |||
} | |||
|
|||
if p.config.InlineShebang == "" { | |||
p.config.InlineShebang = "/bin/sh -e" | |||
if p.config.Inline != nil && len(p.config.Inline) > 0 && strings.HasPrefix(p.config.Inline[0], "#!") { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'd think you can drop the p.config.Inline != nil
since len(nil)
is 0, so the second check alone is fine there
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also I would suggest moving that to the Provision
function, as there could be some interpolation in the string here, which depending on the source of the data, could be something not yet known during the Prepare
step. I don't have an example with that in mind (realistically it seems unlikely), but out of caution I'd think it's better to move that to when the final script is being created.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I have moved this to provision, and now I check for shebang only once the interpolation has taken place. If the user has defined InlineShebang
then we take this value in any case.
provisioner/shell/provisioner.go
Outdated
newLineIndex = len(p.config.Inline[0]) | ||
} | ||
firstLine := strings.TrimRight(p.config.Inline[0][:newLineIndex], "\r") | ||
p.config.InlineShebang = strings.TrimPrefix(firstLine, "#!") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Instead of setting the InlineShebang
for this, I would advise leaving the default behaviour with it, and do nothing if the inline script starts with a shebang, otherwise we'll end-up with the shebang twice in the script. While this is not strictly problematic, it's a bit of a sore imho, doing nothing and leaving the script in the original state seems more indicated.
Though I imagine this might clash with the shebang if we leave it as the default, so maybe we should check that it's not the default value during Provision
instead of checking if it's the empty string.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, I have removed the part where we extract the shebang from the script. Now the script is intact if it has a shebang, we don't inject ours.
Also, I have added a boolean inlineShebangDefined
which gets set if user has defined a custom shebang. Imho this is a bit cleaner than comparing the value of default shebang.
// Test with a inline that does not have a shebang | ||
config["inline"] = []interface{}{"foo", "bar"} | ||
delete(config, "inline_shebang") | ||
p = new(Provisioner) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Given that the logic is duplicated here, I would suggest you run those as subtests, something like https://github.com/hashicorp/packer/blob/main/hcl2template/function/datetime_test.go#L35
Although the other tests in this file seem to adopt the same pattern as yours, so we could also keep it that way and change it later
Test cases will fail as expected since |
provisioner/shell/provisioner.go
Outdated
@@ -203,13 +208,22 @@ func (p *Provisioner) Provision(ctx context.Context, ui packersdk.Ui, comm packe | |||
|
|||
// Write our contents to it | |||
writer := bufio.NewWriter(tf) | |||
_, _ = writer.WriteString(fmt.Sprintf("#!%s\n", p.config.InlineShebang)) | |||
var shebangWritten bool |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'd think you can do the check here and keep the same kind of logic for injecting the shebang to the script, no need for the variable I believe.
var shebangWritten bool | |
if !strings.HasPrefix(p.config.Inline[0], "#!") || p.inlineShebangDefined { | |
writer.WriteString("#!%s\n", p.config.InlineShebang) | |
} |
Though I imagine there might be one case that fails: if the first inline entry is an empty string.
Example:
provisioner "shell" {
inline = ["", "#!/bin/bash", "echo 'test'"]
}
Admittedly this would be weird to have something like that for any reason other than throw off the logic, besides even with the current logic this would fail so I'm not sure we need to burden ourselves with that. If we wanted to address that, I would maybe suggest then we try adding the shebang once all the script has been generated from the individual parts, but I want to stress that this is mostly a theoretical problem, I would be surprised we ran into this for reasons other than just throwing this logic off.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The shebangWritten
bool helps to write the shebang only once and only during the first command. This helps in the case when the user has defined inline_shebang
in template and p.inlineShebangDefined
would hold true for all the commands, resulting in multiple shebangs. Although as a hack, we can make p.inlineShebangDefined = false
once its written, but the variable no longer holds the right information which I don't prefer doing.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
To be clear, my suggestion is to move the logic that injects the shebang outside of the loop, kinda like how it was done before, in this case you don't need to manipulate p.config.inlineShebangDefined
, and the variable is not needed as well since there's a single injection site, outside of the loop that builds the script from its individual parts
@@ -98,6 +98,50 @@ func TestProvisionerPrepare_InlineShebang(t *testing.T) { | |||
} | |||
} | |||
|
|||
func TestProvisionerPrepare_ShebangWithinInline(t *testing.T) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Given this is to be superseded by the packer_test implementation for testing the provisioner, I'd think these can be removed
provisioner/shell/provisioner.go
Outdated
@@ -66,6 +66,9 @@ type Config struct { | |||
// name of the tmp environment variable file, if UseEnvVarFile is true | |||
envVarFile string | |||
|
|||
// If user provided a shebang for inline scripts |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The way I understand this attribute is just so we don't look at the value for InlineShebang
during Provision
, is that right?
I would assume that we don't strictly need it, since we can rely on the default value for InlineShebang
and assume it's the default value, but I can see a world where someone wants to explicitly add the "default" shebang, even if there's already one defined in the script.
I would maybe suggest adding more context to why that attribute is relevant so we don't ask ourselves the question in the future.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, you're right. Its to make sure the user wanted the InlineShebang
in the script.
If not for the default value handling, even I wouldn't want this extra variable in here. But ig its not too bad, I will add more context!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I have one question if the inlineShebang is passed in the inlineShebang object we are expecting it /bin/bash -ex
way and when it is passed in the inline commands, in the first line we are expecting it to be #!/bin/bash -ex
, don't you see inconsistency here ?
@anshulsharma-hashicorp Yes there is, but I'm guessing this has been there for quite a bit and fixing this now could be a breaking change. We can maybe take this up later as a separate issue. |
@anurag5sh okay in that case let it be this way, we can definitely look into it later as a separate issue. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi @anurag5sh,
We're on the right path, but I think we can still improve this code a bit more, I left a couple of comments, feel free to address them when you can, and I'll do another pass of review after that.
// Or If command does not start with a shebang, use the default shebang. | ||
// else command already has a shebang, so do not write it. | ||
if p.config.inlineShebangDefined || !strings.HasPrefix(firstLine, "#!") { | ||
if _, err := fileWriter.WriteString(fmt.Sprintf("#!%s\n", p.config.InlineShebang)); err != nil { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You can use fmt.Fprintf
here instead of filewriter.WriteString(fmt.Sprintf
if _, err := fileWriter.WriteString(fmt.Sprintf("#!%s\n", p.config.InlineShebang)); err != nil { | |
if _, err := fmt.Fprintf(fileWriter, "#!%s\n", p.config.InlineShebang); err != nil { |
return fmt.Errorf("Error preparing shell script: %s", err) | ||
} | ||
} | ||
|
||
if err := writer.Flush(); err != nil { | ||
firstLine, err := commandBuffer.ReadString('\n') |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You don't need to read the first line I'd think, instead you can probably use String()
from the buffer you created, if it's a strings.Builder
, the operation is virtually free, so no need to perform this extra computation.
writer := bufio.NewWriter(tf) | ||
_, _ = writer.WriteString(fmt.Sprintf("#!%s\n", p.config.InlineShebang)) | ||
fileWriter := bufio.NewWriter(tf) | ||
commandBuffer := &bytes.Buffer{} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This could be a strings.Builder
here instead of a bytes.Buffer
so the conversion to a string is free later on.
@@ -202,20 +210,38 @@ func (p *Provisioner) Provision(ctx context.Context, ui packersdk.Ui, comm packe | |||
scripts = append(scripts, tf.Name()) | |||
|
|||
// Write our contents to it | |||
writer := bufio.NewWriter(tf) | |||
_, _ = writer.WriteString(fmt.Sprintf("#!%s\n", p.config.InlineShebang)) | |||
fileWriter := bufio.NewWriter(tf) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would suggest moving the logic that creates the file and its writer to after the loop that feeds the temporary buffer, so the file is only created if there are no problems with the command's creation.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Come to think of it, do we really need a bufio.NewWriter
here?
I would think we can use tf
directly to write to the file, no need to have an extra buffer in between, we likely don't have enough data to write for it to make a noticeable difference, and we can remove a layer of indirection this way.
for _, command := range p.config.Inline { | ||
p.config.ctx.Data = generatedData | ||
command, err := interpolate.Render(command, &p.config.ctx) | ||
if err != nil { | ||
return fmt.Errorf("Error interpolating Inline: %s", err) | ||
} | ||
if _, err := writer.WriteString(command + "\n"); err != nil { | ||
if _, err := commandBuffer.WriteString(command + "\n"); err != nil { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since we're iterating on this code, I would also suggest moving to use a fmt.Fprintf
here so we don't do the extra concatenation needlessly.
if _, err := commandBuffer.WriteString(command + "\n"); err != nil { | |
if _, err := fmt.Fprintf(commandBuffer, "%s\n", command); err != nil { |
} | ||
|
||
// Write the collected commands to the file buffer | ||
if _, err := fileWriter.Write(commandBuffer.Bytes()); err != nil { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You can probably move to use an io.Copy
here, it's likely similar in terms of performance, but might be a bit better.
if _, err := fileWriter.Write(commandBuffer.Bytes()); err != nil { | |
if _, err := io.Copy(fileWriter, commandBuffer); err != nil { |
return fmt.Errorf("Error preparing shell script: %s", err) | ||
} | ||
} | ||
|
||
if err := writer.Flush(); err != nil { | ||
firstLine, err := commandBuffer.ReadString('\n') | ||
if err != nil && err != io.EOF { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The io.EOF
check here seems superfluous, since you're reading from the first bytes, and we always add a line feed after each inline
info, we won't encounter this I'd think.
Though maybe if inline = []
? Might be good to add that to the test suite to make sure it doesn't break, it would be unlikely to encounter a case like this one, but can't hurt.
Regarding the |
UPDATE: Instead of adding a new
content
field, the existinginline
field is enhanced to support any script with its own shebang.The behavior of inline is mentioned here - #13313 (comment)
Closes #11437
Added thecontent
field to the shell provisioner to accept any string(supports output oftemplatefile
) as a script and execute it. This is in line with thecontent
field of the file provisioner.With this change, the shell provisioner now supports 4 different ways in which the cmd/script can be executed in target system. Exactly one of these are accepted:inline
,script
,scripts
orcontent
.I have modified the conditonal checks to ensure only one of the above 4 values are accepted.Added the same unit tests that were raised in the previous PR.
Test configuration files:
test.pkr.hcl
script.sh.tmpl
> packer build test.pkr.hcl