Passing additional parameters specific to efficient apriori does not seem to work outside of the 4 parameters mentioned

[DDekov]

Function
Efficient-Apriori

Describe the bug
I was hoping that the SSE would have a way to pass additional parameters to the efficient-apriori outside of the 4 parameters mentioned in the documentation. Not sure if this a bug or a feature request or a documentation/help request. What I am trying to do is to use the parameter output_transaction_ids as described here: https://github.com/tommyod/Efficient-Apriori. It is important to have this parameter in order to be able to have a meaningful link between the association rules and the data set. The current examples of implementing the apriori do not have a proper link to the transaction as linking through the RHE and LHE parts of the rule brings you links from the left part or right part of the rule, but that is not a link from the whole rule. The effect is a very non-Qlik experience in the dashboard front end. I think having the Rule->Transaction IDs link is vital.

Full error message from Python
Provide full stack trace for the error.
If using Qlik-Py-Start you can provide screenshots from the terminal. Make sure to go all the way up to the start of the errors.
Alternatively, you can provide the latest log files from ../qlik-py-tools/qlik-py-env/core/logs.

Error message in Qlik

Expression or load script used in Qlik
The Qlik script is standard - based on the demos provided with the SSE. The script is running OK, the problem is either in the SSE or in the python environment. Here is the important part of the script:

Environment:

• OS Windows Server 2012 R2
• Qlik Sense Enterprise June 2020
• SSE Version: 8.1

Additional context
Add any other context about the problem here.

[DDekov]

After digging a bit in the code of the efficient apriori library, it seems that the output_transaction_ids parameter was introduced in later versions. The 1.0.0 version just doesn't have the capability. The SSE seems to be properly passing the parameters to efficient apriori, but in 1.0.0 the output_transaction_ids parameter is just not a valid one. The solution will be to upgrade to version 1.1.1. I am not doing the upgrade along with some dependencies and will share the results after testing.

[DDekov]

Here is the fix to the Qlik-Py-Init.bat: we need to add these lines to make the apriori 1.1.1 work:

[DDekov]

now the last problem I am tackling seems to be with the SSE (I might be wrong) - python is returning the table with the added field, but Qlik doesn't take it in the load. @nabeel-oz your help might be needed here.

[nabeel-oz]

Hi @DDekov , This seems to be a new feature of Efficient-Apriori as you mentioned and is not implemented by the SSE. Implementing this capability would require the transaction IDs returned by Efficient-Apriori to be mapped to the data from Qlik and then returned in a suitable format. I'll take this into consideration for a future release, but am not planning to work on this in the short term.

However, I think you can model the required association using standard Qlik load script. For example, I have attached a version of the Market Basket Analysis sample app that links the Association Rules to Invoice Numbers instead of LHS Products.

https://www.dropbox.com/s/t6botk234fpxh9g/Sample%20App%20-%20Market%20Basket%20Analysis.qvf?dl=1

This just required an additional section in the load script of the original app.

// Concatenate unique products in each invoice
[Invoice Products Temp]:
LOAD Distinct
    InvoiceNo,
    Concat(Distinct Product, ';', Product) as InvoiceProducts
RESIDENT [Online Retail]
GROUP BY InvoiceNo;

// Concatenate products in each association rule
[Rule Product Keys]:
LOAD Distinct
    [Association Rule],
    Concat(Distinct AR_Product, ';', AR_Product) as RuleProducts,
    '*' & Concat(Distinct AR_Product, '*', AR_Product) & '*' as RuleProductKey // Set up wildmatch search text
RESIDENT [Rule Products]
GROUP BY [Association Rule];

// Set up an empty table that will be populated through a loop
[Invoice Rule Link]:
LOAD * INLINE [
	InvoiceNo, InvoiceProducts, RuleProductKey
];

// Loop through products from the rules and match against invoices
FOR EACH vRuleProductKey in FieldValueList('RuleProductKey')

    Concatenate([Invoice Rule Link])
    LOAD
    	InvoiceNo,
        InvoiceProducts,
        '$(vRuleProductKey)' as RuleProductKey
    RESIDENT [Invoice Products Temp]
    WHERE WildMatch(InvoiceProducts, '$(vRuleProductKey)') > 0; // Check that all products from the rule are in the invoice

NEXT vRuleProductKey

LEFT JOIN([Invoice Rule Link])
LOAD
    RuleProductKey,
    [Association Rule],
    RuleProducts    
RESIDENT [Rule Product Keys];

Drop table [Invoice Products Temp], [Rule Product Keys];

[DDekov]

Quite a neat solution, @nabeel-oz. I was actually working on something similar, but went a bit in the wrong way of creating all permutations of the rule items and then doing the search. No permutations are needed if you just do the distinct and sort them. Nice and neat!

[nabeel-oz]

Thanks! Glad the solution works for you.