Skip to content

Commit

Permalink
Merge branch 'develop-template-vars' into develop. Close #250.
Browse files Browse the repository at this point in the history
**Description**

The generated ROS, FPrime and standalone application may need definitions that
come from external files, such as topic names from other applications, or
structs defined elsewhere.

Users may also want to customize additional aspects of the generated
applications without having to modify the template, such as passing additional
variables to be expanded in the template.

There should be a setting to expand other variables so that the generated app
never has to be modified.

**Type**

- Feature: facilitate customizing generated applications.

**Additional context**

None.

**Requester**

- Ivan Perez.

**Method to check presence of bug**

Not applicable (not a bug).

**Expected result**

Ogma provides a mechanism to provide additional variables to be expanded in
ROS, FPrime and standalone applications.

The following dockerfile first checks that passing the additional extra
variables files to different backends does not affect the result unless the
variable is mentioned in the template, by comparing the result to a run without
passing any extra vars. It later adds a file to each custom template mentioning
a new variable, and then passes an extra template variable using the new flag,
after which it prints the message "Success" 5 times (one for each combination
of backends and options tried), indicating that the new variable was expanded
as expected:

```
--- Dockerfile-250
FROM ubuntu:focal

ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update

RUN apt-get install --yes \
      curl g++ gcc git libgmp3-dev libz-dev make pkg-config

RUN mkdir -p $HOME/.local/bin
ENV PATH=$PATH:/root/.local/bin/

RUN curl https://downloads.haskell.org/~ghcup/0.1.17.7/x86_64-linux-ghcup-0.1.17.7 -o $HOME/.local/bin/ghcup
RUN chmod a+x $HOME/.local/bin/ghcup

ENV PATH=$PATH:/root/.ghcup/bin/
RUN ghcup install ghc 9.10
RUN ghcup install cabal 3.12
RUN ghcup set ghc 9.10.1
RUN cabal update

ADD document.json      /tmp/document.json
ADD json-format.cfg    /tmp/json-format.cfg
ADD fprime-variable-db /tmp/fprime-variable-db
ADD fprime-handlers    /tmp/fprime-handlers

SHELL ["/bin/bash", "-c"]
CMD git clone $REPO \
    && cd $NAME \
    && git checkout $COMMIT \
    && cabal install ogma-cli:ogma \
    && ogma fprime     --app-target-dir fprime_original        --handlers-file /tmp/fprime-handlers                   --input-file /tmp/document.json -f /tmp/json-format.cfg -p literal --variable-db /tmp/fprime-variable-db \
    && ogma ros        --app-target-dir ros_spec_original      --handlers-file ogma-cli/examples/ros-copilot/handlers --input-file /tmp/document.json -f /tmp/json-format.cfg -p literal --variable-db ogma-cli/examples/ros-copilot/vars-db \
    && ogma ros        --app-target-dir ros_variables_original --handlers-file ogma-cli/examples/ros-copilot/handlers --variable-db ogma-cli/examples/ros-copilot/vars-db     --variable-file ogma-cli/examples/ros-copilot/variables \
    && ogma cfs        --app-target-dir cfs_original           --handlers-file ogma-cli/examples/cfs-handlers         --variable-db ogma-cli/examples/cfs-variable-db         --variable-file ogma-cli/examples/cfs-variables \
    && ogma standalone --target-dir     standalone_original    --file-name /tmp/document.json -f /tmp/json-format.cfg -p literal --target-file-name copilot \
    && cp -r ogma-core/templates custom \
    && find custom -iname '.gitignore' -delete \
    && echo '{ "message": "Success" }' >> template-vars \
    && ogma fprime     --app-template-dir custom/fprime      --template-vars template-vars --app-target-dir fprime_updated         --handlers-file /tmp/fprime-handlers                   --input-file /tmp/document.json -f /tmp/json-format.cfg -p literal --variable-db /tmp/fprime-variable-db \
    && ogma ros        --app-template-dir custom/ros         --template-vars template-vars --app-target-dir ros_spec_updated       --handlers-file ogma-cli/examples/ros-copilot/handlers --input-file /tmp/document.json -f /tmp/json-format.cfg -p literal --variable-db ogma-cli/examples/ros-copilot/vars-db \
    && ogma ros        --app-template-dir custom/ros         --template-vars template-vars --app-target-dir ros_variables_updated  --handlers-file ogma-cli/examples/ros-copilot/handlers --variable-db ogma-cli/examples/ros-copilot/vars-db  --variable-file ogma-cli/examples/ros-copilot/variables \
    && ogma cfs        --app-template-dir custom/copilot-cfs --template-vars template-vars --app-target-dir cfs_updated            --handlers-file ogma-cli/examples/cfs-handlers         --variable-db ogma-cli/examples/cfs-variable-db      --variable-file ogma-cli/examples/cfs-variables \
    && ogma standalone --template-dir custom/standalone  --template-vars template-vars --target-dir     standalone_updated     --file-name /tmp/document.json -f /tmp/json-format.cfg -p literal --target-file-name copilot \
    && diff -rq fprime_original        fprime_updated \
    && diff -rq ros_variables_original ros_variables_updated \
    && diff -rq ros_spec_original      ros_spec_updated \
    && diff -rq cfs_original           cfs_updated \
    && diff -rq standalone_original    standalone_updated \
    && rm -rf *_updated \
    && echo '{{message}}' >> custom/copilot-cfs/fsw/result \
    && echo '{{message}}' >> custom/ros/result \
    && echo '{{message}}' >> custom/fprime/result \
    && echo '{{message}}' >> custom/standalone/result \
    && ogma fprime     --app-template-dir custom/fprime      --template-vars template-vars --app-target-dir fprime_updated         --handlers-file /tmp/fprime-handlers                   --input-file /tmp/document.json -f /tmp/json-format.cfg -p literal --variable-db /tmp/fprime-variable-db \
    && ogma ros        --app-template-dir custom/ros         --template-vars template-vars --app-target-dir ros_spec_updated       --handlers-file ogma-cli/examples/ros-copilot/handlers --input-file /tmp/document.json -f /tmp/json-format.cfg -p literal --variable-db ogma-cli/examples/ros-copilot/vars-db \
    && ogma ros        --app-template-dir custom/ros         --template-vars template-vars --app-target-dir ros_variables_updated  --handlers-file ogma-cli/examples/ros-copilot/handlers --variable-db ogma-cli/examples/ros-copilot/vars-db  --variable-file ogma-cli/examples/ros-copilot/variables \
    && ogma cfs        --app-template-dir custom/copilot-cfs --template-vars template-vars --app-target-dir cfs_updated            --handlers-file ogma-cli/examples/cfs-handlers         --variable-db ogma-cli/examples/cfs-variable-db      --variable-file ogma-cli/examples/cfs-variables \
    && ogma standalone --template-dir custom/standalone  --template-vars template-vars --target-dir     standalone_updated     --file-name /tmp/document.json -f /tmp/json-format.cfg -p literal --target-file-name copilot \
    && cat fprime_updated/result \
    && cat ros_spec_updated/result \
    && cat ros_variables_updated/result \
    && cat cfs_updated/fsw/result \
    && cat standalone_updated/result \
    && echo "Complete Success"

--- document.json
{
  "Example": {
    "internal_variables": [],
    "external_variables": [
      {"name":"input", "type":"Float", "meaning": "Input from the system"}
    ],
    "properties": [
      {
        "id":      "MyProperty",
        "formula": "PTLTL.alwaysBeen (input /= 30.0)",
        "text":    "Input is never 30"
      }
    ]
  }
}

--- fprime-handlers
handlerMyProperty

--- fprime-variable-db
("pullup", "bool")
("input", "float")

--- json-format.cfg
JSONFormat
   { specInternalVars    = Just "..internal_variables[*]"
   , specInternalVarId   = ".name"
   , specInternalVarExpr = ".meaning"
   , specInternalVarType = Just ".type"
   , specExternalVars    = Just "..external_variables[*]"
   , specExternalVarId   = ".name"
   , specExternalVarType = Just ".type"
   , specRequirements    = "..properties[*]"
   , specRequirementId   = ".id"
   , specRequirementDesc = Just ".text"
   , specRequirementExpr = ".formula"
   }
```

Command (substitute variables based on new path after merge):
```sh
$ docker run -e "REPO=https://github.com/NASA/ogma" -e "NAME=ogma" -e "COMMIT=<HASH>" -it ogma-verify-250
```

**Solution implemented**

Expand the ROS, FPrime and standalone backends in `ogma-core` to expect and use
an additional optional argument pointing to a JSON file with additional
variables to expand in the template.

Adjust tests in `ogma-core` based on the new API.

Provide a CLI flag for the ROS, FPrime and standalone backends that allows
filling in additional template variables.

Document new flag in the README.

**Further notes**

None.
  • Loading branch information
ivanperez-keera committed Feb 10, 2025
2 parents 5b6f5ac + 8a3d4ea commit 8576131
Show file tree
Hide file tree
Showing 10 changed files with 266 additions and 44 deletions.
3 changes: 2 additions & 1 deletion ogma-cli/CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Revision history for ogma-cli

## [1.X.Y] - 2025-02-08
## [1.X.Y] - 2025-02-09

* Add all auxiliary test files to distributable Cabal package (#216).
* Remove extraneous EOL character (#224).
Expand All @@ -13,6 +13,7 @@
* Update README with new ROS template variables (#244).
* Update README with new FPrime template variables (#246).
* Adjust CLI to match new backend API (#248).
* Expose template-vars argument to ROS, FPrime, standalone backends (#250).

## [1.6.0] - 2025-01-21

Expand Down
4 changes: 4 additions & 0 deletions ogma-cli/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -282,6 +282,8 @@ be made available to the monitor.
and the topic they are included with.
- `--handlers FILENAME`: a file containing a list of handlers used in the
specification.
- `--template-vars FILENAME`: a JSON file containing a list of additional
variables to expand in the template.
The following execution generates an initial ROS application for runtime
monitoring using Copilot:
Expand Down Expand Up @@ -405,6 +407,8 @@ be made available to the monitor.
and their types.
- `--handlers FILENAME`: a file containing a list of handlers used in the
specification.
- `--template-vars FILENAME`: a JSON file containing a list of additional
variables to expand in the template.

The following execution generates an initial F' component for runtime
monitoring using Copilot:
Expand Down
32 changes: 23 additions & 9 deletions ogma-cli/src/CLI/CommandFPrimeApp.hs
Original file line number Diff line number Diff line change
Expand Up @@ -57,15 +57,16 @@ import qualified Command.FPrimeApp

-- | Options needed to generate the FPrime component.
data CommandOpts = CommandOpts
{ fprimeAppInputFile :: Maybe String
, fprimeAppTarget :: String
, fprimeAppTemplateDir :: Maybe String
, fprimeAppVariables :: Maybe String
, fprimeAppVarDB :: Maybe String
, fprimeAppHandlers :: Maybe String
, fprimeAppFormat :: String
, fprimeAppPropFormat :: String
, fprimeAppPropVia :: Maybe String
{ fprimeAppInputFile :: Maybe String
, fprimeAppTarget :: String
, fprimeAppTemplateDir :: Maybe String
, fprimeAppVariables :: Maybe String
, fprimeAppVarDB :: Maybe String
, fprimeAppHandlers :: Maybe String
, fprimeAppFormat :: String
, fprimeAppPropFormat :: String
, fprimeAppPropVia :: Maybe String
, fprimeAppTemplateVars :: Maybe String
}

-- | Create <https://github.com/nasa/fprime FPrime> component that subscribe
Expand All @@ -87,6 +88,7 @@ command c = Command.FPrimeApp.command options
, Command.FPrimeApp.commandFormat = fprimeAppFormat c
, Command.FPrimeApp.commandPropFormat = fprimeAppPropFormat c
, Command.FPrimeApp.commandPropVia = fprimeAppPropVia c
, Command.FPrimeApp.commandExtraVars = fprimeAppTemplateVars c
}

-- * CLI
Expand Down Expand Up @@ -164,6 +166,13 @@ commandOptsParser = CommandOpts
<> help strFPrimeAppPropViaDesc
)
)
<*> optional
( strOption
( long "template-vars"
<> metavar "FILENAME"
<> help strFPrimeAppTemplateVarsArgDesc
)
)

-- | Argument target directory to FPrime component generation command
strFPrimeAppDirArgDesc :: String
Expand Down Expand Up @@ -206,3 +215,8 @@ strFPrimeAppPropFormatDesc = "Format of temporal or boolean properties"
strFPrimeAppPropViaDesc :: String
strFPrimeAppPropViaDesc =
"Command to pre-process individual properties"

-- | Additional template variable file flag description.
strFPrimeAppTemplateVarsArgDesc :: String
strFPrimeAppTemplateVarsArgDesc =
"JSON file containing additional variables to expand in template"
32 changes: 23 additions & 9 deletions ogma-cli/src/CLI/CommandROSApp.hs
Original file line number Diff line number Diff line change
Expand Up @@ -57,15 +57,16 @@ import qualified Command.ROSApp

-- | Options needed to generate the ROS application.
data CommandOpts = CommandOpts
{ rosAppInputFile :: Maybe String
, rosAppTarget :: String
, rosAppTemplateDir :: Maybe String
, rosAppVarNames :: Maybe String
, rosAppVarDB :: Maybe String
, rosAppHandlers :: Maybe String
, rosAppFormat :: String
, rosAppPropFormat :: String
, rosAppPropVia :: Maybe String
{ rosAppInputFile :: Maybe String
, rosAppTarget :: String
, rosAppTemplateDir :: Maybe String
, rosAppVarNames :: Maybe String
, rosAppVarDB :: Maybe String
, rosAppHandlers :: Maybe String
, rosAppFormat :: String
, rosAppPropFormat :: String
, rosAppPropVia :: Maybe String
, rosAppTemplateVars :: Maybe String
}

-- | Create <https://www.ros.org/ Robot Operating System> (ROS) applications
Expand All @@ -86,6 +87,7 @@ command c = Command.ROSApp.command options
, Command.ROSApp.commandFormat = rosAppFormat c
, Command.ROSApp.commandPropFormat = rosAppPropFormat c
, Command.ROSApp.commandPropVia = rosAppPropVia c
, Command.ROSApp.commandExtraVars = rosAppTemplateVars c
}

-- * CLI
Expand Down Expand Up @@ -163,6 +165,13 @@ commandOptsParser = CommandOpts
<> help strROSAppPropViaDesc
)
)
<*> optional
( strOption
( long "template-vars"
<> metavar "FILENAME"
<> help strROSAppTemplateVarsArgDesc
)
)

-- | Argument target directory to ROS app generation command
strROSAppDirArgDesc :: String
Expand Down Expand Up @@ -205,3 +214,8 @@ strROSAppPropFormatDesc = "Format of temporal or boolean properties"
strROSAppPropViaDesc :: String
strROSAppPropViaDesc =
"Command to pre-process individual properties"

-- | Additional template variable file flag description.
strROSAppTemplateVarsArgDesc :: String
strROSAppTemplateVarsArgDesc =
"JSON file containing additional variables to expand in template"
30 changes: 22 additions & 8 deletions ogma-cli/src/CLI/CommandStandalone.hs
Original file line number Diff line number Diff line change
Expand Up @@ -58,14 +58,15 @@ import qualified Command.Standalone

-- | Options to generate Copilot from specification.
data CommandOpts = CommandOpts
{ standaloneTargetDir :: FilePath
, standaloneTemplateDir :: Maybe FilePath
, standaloneFileName :: FilePath
, standaloneFormat :: String
, standalonePropFormat :: String
, standaloneTypes :: [String]
, standaloneTarget :: String
, standalonePropVia :: Maybe String
{ standaloneTargetDir :: FilePath
, standaloneTemplateDir :: Maybe FilePath
, standaloneFileName :: FilePath
, standaloneFormat :: String
, standalonePropFormat :: String
, standaloneTypes :: [String]
, standaloneTarget :: String
, standalonePropVia :: Maybe String
, standaloneTemplateVars :: Maybe String
}

-- | Transform an input specification into a Copilot specification.
Expand All @@ -83,6 +84,7 @@ command c =
, Command.Standalone.commandTypeMapping = types
, Command.Standalone.commandFilename = standaloneTarget c
, Command.Standalone.commandPropVia = standalonePropVia c
, Command.Standalone.commandExtraVars = standaloneTemplateVars c
}

types :: [(String, String)]
Expand Down Expand Up @@ -161,6 +163,13 @@ commandOptsParser = CommandOpts
<> help strStandalonePropViaDesc
)
)
<*> optional
( strOption
( long "template-vars"
<> metavar "FILENAME"
<> help strStandaloneTemplateVarsArgDesc
)
)

-- | Target dir flag description.
strStandaloneTargetDirDesc :: String
Expand Down Expand Up @@ -195,3 +204,8 @@ strStandaloneTargetDesc =
strStandalonePropViaDesc :: String
strStandalonePropViaDesc =
"Command to pre-process individual properties"

-- | Additional template variable file flag description.
strStandaloneTemplateVarsArgDesc :: String
strStandaloneTemplateVarsArgDesc =
"JSON file containing additional variables to expand in template"
3 changes: 2 additions & 1 deletion ogma-core/CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Revision history for ogma-core

## [1.X.Y] - 2025-02-08
## [1.X.Y] - 2025-02-09

* Import liftIO from Control.Monad.IO.Class (#215).
* Remove references to old design of Ogma from hlint files (#220).
Expand All @@ -17,6 +17,7 @@
* Make structured data available to ROS template (#244).
* Make structured data available to FPrime template (#246).
* Equalize backends (#248).
* Update ROS, FPrime, standalone backends to process template vars file (#250).

## [1.6.0] - 2025-01-21

Expand Down
67 changes: 62 additions & 5 deletions ogma-core/src/Command/FPrimeApp.hs
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,9 @@ import qualified Control.Exception as E
import Control.Monad.Except ( ExceptT(..), liftEither, runExceptT,
throwError )
import Control.Monad.IO.Class ( liftIO )
import Data.Aeson ( ToJSON, eitherDecode, object, toJSON )
import Data.Aeson ( ToJSON, Value (Null, Object),
eitherDecode, object, toJSON )
import Data.Aeson.KeyMap ( union )
import Data.Char ( toUpper )
import Data.List ( isInfixOf, isPrefixOf, find )
import Data.Maybe ( fromMaybe, mapMaybe )
Expand Down Expand Up @@ -95,19 +97,22 @@ command options = processResult $ do
-- Obtain template dir
templateDir <- locateTemplateDir mTemplateDir "fprime"

templateVars <- parseTemplateVarsFile templateVarsF

appData <- command' options functions

let subst = toJSON appData
let subst = mergeObjects (toJSON appData) templateVars

-- Expand template
ExceptT $ fmap (makeLeftE cannotCopyTemplate) $ E.try $
copyTemplate templateDir subst targetDir

where

targetDir = commandTargetDir options
mTemplateDir = commandTemplateDir options
functions = exprPair (commandPropFormat options)
targetDir = commandTargetDir options
mTemplateDir = commandTemplateDir options
functions = exprPair (commandPropFormat options)
templateVarsF = commandExtraVars options

command' :: CommandOptions
-> ExprPair
Expand Down Expand Up @@ -165,6 +170,9 @@ data CommandOptions = CommandOptions
, commandPropFormat :: String -- ^ Format used for input properties.
, commandPropVia :: Maybe String -- ^ Use external command to
-- pre-process system properties.
, commandExtraVars :: Maybe FilePath -- ^ File containing additional
-- variables to make available to the
-- template.
}

-- | Return the variable information needed to generate declarations
Expand Down Expand Up @@ -318,6 +326,20 @@ parseVarDBFile (Just fn) =
ExceptT $ makeLeftE (cannotOpenDB fn) <$>
(E.try $ fmap read <$> lines <$> readFile fn)

-- | Process a JSON file with additional template variables to make available
-- during template expansion.
parseTemplateVarsFile :: Maybe FilePath
-> ExceptT ErrorTriplet IO Value
parseTemplateVarsFile Nothing = return $ object []
parseTemplateVarsFile (Just fp) = do
content <- liftIO $ B.safeReadFile fp
let value = eitherDecode =<< content
case value of
Right x@(Object _) -> return x
Right x@Null -> return x
Right _ -> throwError (cannotReadObjectTemplateVars fp)
_ -> throwError (cannotOpenTemplateVars fp)

-- | Check that the arguments provided are sufficient to operate.
--
-- The backend provides several modes of operation, which are selected
Expand Down Expand Up @@ -497,6 +519,24 @@ commandIncorrectFormatSpec formatFile =
"The format specification " ++ formatFile ++ " does not exist or is not "
++ "readable"

-- | Exception handler to deal with the case in which the template vars file
-- cannot be opened.
cannotOpenTemplateVars :: FilePath -> ErrorTriplet
cannotOpenTemplateVars file =
ErrorTriplet ecCannotOpenTemplateVarsFile msg (LocationFile file)
where
msg =
"Cannot open file with additional template variables: " ++ file

-- | Exception handler to deal with the case in which the template vars file
-- cannot be opened.
cannotReadObjectTemplateVars :: FilePath -> ErrorTriplet
cannotReadObjectTemplateVars file =
ErrorTriplet ecCannotReadObjectTemplateVarsFile msg (LocationFile file)
where
msg =
"Cannot open file with additional template variables: " ++ file

-- | Exception handler to deal with the case of files that cannot be
-- copied/generated due lack of space or permissions or some I/O error.
cannotCopyTemplate :: ErrorTriplet
Expand Down Expand Up @@ -534,6 +574,14 @@ ecCannotOpenHandlersFile = 1
ecIncorrectFormatFile :: ErrorCode
ecIncorrectFormatFile = 1

-- | Error: the template vars file provided by the user cannot be opened.
ecCannotOpenTemplateVarsFile :: ErrorCode
ecCannotOpenTemplateVarsFile = 1

-- | Error: the template variables file passed does not contain a JSON object.
ecCannotReadObjectTemplateVarsFile :: ErrorCode
ecCannotReadObjectTemplateVarsFile = 1

-- | Error: the files cannot be copied/generated due lack of space or
-- permissions or some I/O error.
ecCannotCopyTemplate :: ErrorCode
Expand All @@ -552,6 +600,15 @@ locateTemplateDir mTemplateDir name =
dataDir <- getDataDir
return $ dataDir </> "templates" </> name

-- | Merge two JSON objects.
--
-- Fails if the values are not objects or null.
mergeObjects :: Value -> Value -> Value
mergeObjects (Object m1) (Object m2) = Object (union m1 m2)
mergeObjects obj Null = obj
mergeObjects Null obj = obj
mergeObjects _ _ = error "The values passed are not objects"

-- | Replace the left Exception in an Either.
makeLeftE :: c -> Either E.SomeException b -> Either c b
makeLeftE c (Left _) = Left c
Expand Down
Loading

0 comments on commit 8576131

Please sign in to comment.