Incomplete CLI documentation
Forum
  1. Forums
  2. Imunify360
  3. Imunify360 and Imunify Sensor
  1. John Carpenter
  2. 19.02.2020
  3.  Subscribe via email
Hello,

I've noticed a huge inconsistency between the number of CLI commands listed in the documentation, and those reported by command-line help in imunify360-agent output.

For example the documentation lists about 30 top-level commands at https://docs.imunify360.com/command_line_interface/. However the CLI output lists about 50 available commands as "positional arguments" and only 27 "Available commands:"


# imunify360-agent -h
usage: imunify360-agent [-h] [--log-config LOG_CONFIG]
[--console-log-level {ERROR,WARNING,INFO,DEBUG}]
[--remote-addr REMOTE_ADDR]
{3rdparty,add-sudouser,admin-emails,backup-systems,blacklist,blocked-port,blocked-port-ip,check,check-domains,checkdb,clean,config,create-rbl-whitelist,delete-sudouser,disable-plugin,doctor,enable-plugin,eula,feature-management,features,fix,get,get-news,graylist,health,hook,import,infected-domains,install-vendors,login,malware,proactive,register,reload-lists,remote-proxy,remove-block-report-script,remove-csf-ports,restore-configs,rstatus,rules,service,submit,support,uninstall-vendors,unregister,update,update-license,version,whitelist,whitelisted-crawlers}
...

CLI for imunify360 agent.

positional arguments:
{3rdparty,add-sudouser,admin-emails,backup-systems,blacklist,blocked-port,blocked-port-ip,check,check-domains,checkdb,clean,config,create-rbl-whitelist,delete-sudouser,disable-plugin,doctor,enable-plugin,eula,feature-management,features,fix,get,get-news,graylist,health,hook,import,infected-domains,install-vendors,login,malware,proactive,register,reload-lists,remote-proxy,remove-block-report-script,remove-csf-ports,restore-configs,rstatus,rules,service,submit,support,uninstall-vendors,unregister,update,update-license,version,whitelist,whitelisted-crawlers}
Available commands
add-sudouser
admin-emails Get panel admin emails
blacklist
check-domains Send domain list check
checkdb
clean
create-rbl-whitelist
Create whitelist for RBL
delete-sudouser
disable-plugin Disable hosting panel plugin
doctor
enable-plugin Enable hosting panel plugin (if detected)
get
get-news
health
infected-domains Returns infected domain list
install-vendors Enable hosting panel plugin (if detected)
register Register the agent
reload-lists Reload custom black and white lists
remove-block-report-script
Restore block reports script
remove-csf-ports Remove imunify360 ports from csf config
restore-configs Restore system configs to pre install Imunify360 state
rstatus Get registration status
uninstall-vendors Disable hosting panel plugin
unregister Unregister the agent
update
update-license Force update license
version

optional arguments:
-h, --help show this help message and exit
--log-config LOG_CONFIG
logging config filename
--console-log-level {ERROR,WARNING,INFO,DEBUG}
Level of logging input to the console
--remote-addr REMOTE_ADDR
Client's IP address for adding it to the whitelist


To give another example, under the top-level "malware" command, the documentation on your website lists 6 commands. The CLI reports 10 available commands as "positional arguments:"


# imunify360-agent malware -h
usage: imunify360-agent malware [-h]
{cleanup,dashboard,hash,history,ignore,malicious,on-demand,read,suspicious,user}
...

positional arguments:
{cleanup,dashboard,hash,history,ignore,malicious,on-demand,read,suspicious,user}
Available commands
read

optional arguments:
-h, --help show this help message and exit


What is the reason for the discrepancy? Is the documentation still maintained? Should we use commands and features which do not appear in official documentation?

The questions arose because it seems that an on-demand scan started as follows, is not scanning every home directory for reasons unknown:


# imunify360-agent malware on-demand start --path '/home*/*' --no-follow-symlinks --intensity low


However through experimentation, we found that a scan started as follows will apparently scan every user home directory:


# imunify360-agent malware user scan


The latter command seems to work as intended, but I am uneasy using it when it is undocumented. I'm also curious why the single on-demand scan shown above wouldn't include every directory under /home*/*.
Rate this post:
  1. 20.02.2020 08:02:41
  2. # 1
Greg Accepted Answer
Posts: 0
Joined: 30.09.2020
0
Votes
Undo
Hi John,
Thanks for bringing this in. The documentation overall is up-to-date, we're going to check the part related to CLI today and get back to you with the answer. Our product team is on it.

I've checked the command against Imunify360 v4.5

# imunify360-agent malware on-demand start --path '/home*/*' --no-follow-symlinks --intensity low

and it works for me, the scan is queued and run. So I've got a couple of questions:
1. What version of the Imunify360 are you using?
2. What exactly doesn't work (what results you expect and what are the actual results)?

Thanks!
  1. 20.02.2020 14:02:44
  2. # 2
John Carpenter Accepted Answer
Posts: 6
Joined: 28.06.2019
0
Votes
Undo
Hi, thanks for the response.

Version is 4.4.5-1.

The scan seems to start OK but known positives aren't always detected when scanning in this manner. We get many more detections when scanning users individually.

We could scan users individually, but the "malware user scan" feature isn't documented so I was hesitant to use it. I assumed any commands which aren't listed in the documentation wouldn't be considered supported or production ready; was this incorrect? Additionally "malware user scan" doesn't seem to support "--ignore-mask" or "--intensity*" arguments. "--ignore-mask" is especially important for our use case because we're not concerned with scanning mail, cPanel .trash, and some other low risk I/O-intensive locations.

Checking into logs, in /var/log/imunify360/error.log we have the following:


ERROR [2020-02-19 19:17:41,602] defence360agent.malwarelib.scan.scan_result: Scan failed with ScanFailed exception for AiBolit
Traceback (most recent call last):
File "/opt/alt/python35/lib/python3.5/site-packages/defence360agent/malwarelib/scan/ai_bolit.py", line 192, in scan
report = json.loads(out.decode())
File "/opt/alt/python35/lib64/python3.5/json/__init__.py", line 319, in loads
return _default_decoder.decode(s)
File "/opt/alt/python35/lib64/python3.5/json/decoder.py", line 339, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/opt/alt/python35/lib64/python3.5/json/decoder.py", line 357, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 2 column 1 (char 1)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/alt/python35/lib/python3.5/site-packages/defence360agent/malwarelib/scan/scanner.py", line 146, in _run_aibolit
file, progress_cb=self._progress_cb, **scan_options)
File "/opt/alt/python35/lib/python3.5/site-packages/defence360agent/malwarelib/scan/utils.py", line 74, in wrapper
iterator, vendor_dict = await fun(*args, **kwargs)
File "/opt/alt/python35/lib/python3.5/site-packages/defence360agent/malwarelib/scan/utils.py", line 34, in wrapper
iterator, vendor_dict = await fun(*args, **kwargs)
File "/opt/alt/python35/lib/python3.5/site-packages/defence360agent/malwarelib/scan/ai_bolit.py", line 195, in scan
out, err)
defence360agent.malwarelib.scan.ai_bolit.AiBolitError: Scan failed with JSONDecodeError: command: ['/opt/alt/php73/usr/bin/php', '-n', '-d', 'short_open_tag=on', '-d', 'extension=json.so', '-d', 'extension=mbstring.so', '-d', 'extension=leveldb.so', '/opt/ai-bolit/ai-bolit-hoster.php', '--smart', '--deobfuscate', '--avdb', '/var/imunify360/files/sigs/v1/aibolit/ai-bolit-hoster.db', '--no-html', '--json_report', '.', '--json-stdout', '--memory', '1024M', '--listing', '/var/imunify360/tmp/tmpx67_j4n4', '--progress', '/var/imunify360/tmp/ai_bolit_progress_15821602802415218.json', '--with-suspicious', '--size', '1048576', '--cloud-assist', 'IP-77624-319411-C0rGkBUW', '--cloudscan-size', '10485760'] return code: 255 out:
Fatal error: Allowed memory size of 1073741824 bytes exhausted (tried to allocate 4096 bytes) in /opt/ai-bolit/ai-bolit-hoster.php on line 2977
err:


Looks like these errors started on November 19th after successful scans through November 12th:


# imunify360-agent malware on-demand list
CREATED ERROR PATH SCAN_STATUS SCANID STARTED TOTAL TOTAL_FILES TOTAL_MALICIOUS
1582146418 JSONDecodeError /home* stopped 5cc91e7152104edeb53fbc4e52877ad6 1582146418 6631309 6631309 0
1582016527 JSONDecodeError /home*/* stopped f25fa3085e824ede8db1a6aaa381f7d7 1582016527 6631743 6631743 0
1581411727 JSONDecodeError /home*/* stopped 729311affed24fe1ad334d0719273491 1581411727 6594022 6594022 0
1580806941 JSONDecodeError /home*/* stopped c96e46012a4544c781fa6d86316f496f 1580806941 6558293 6558293 0
1580202134 JSONDecodeError /home*/* stopped 8fcbe313f39849c5a738729ec3187cae 1580202134 6490440 6490440 0
1579597326 JSONDecodeError /home*/* stopped 028a32f2667545a3b93e6991d9a4d557 1579597326 6712770 6712770 0
1578387728 JSONDecodeError /home*/* stopped 11540e4ee179453880b577d8f2a9edac 1578387728 6379406 6379406 0
1577782927 JSONDecodeError /home*/* stopped 431f2d37452142478d857f9b179d7eaa 1577782927 6424507 6424507 0
1577178127 JSONDecodeError /home*/* stopped dd5e0d3585474514b3de31ca273bf465 1577178127 6422732 6422732 0
1576573328 JSONDecodeError /home*/* stopped 1bfdc81cf7a343f1992c99cfc23c74d2 1576573328 6396889 6396889 0
1575968527 JSONDecodeError /home*/* stopped a90a9f2b7270448e99311ba64c1e6928 1575968527 6358301 6358301 0
1575363726 JSONDecodeError /home*/* stopped 79bcf9152be6486eb0b0f57e68e4f74e 1575363726 6181119 6181119 0
1574758926 JSONDecodeError /home*/* stopped 2b616f71d1464b5e886b231281cd4d71 1574758926 6192508 6192508 0
1574154126 JSONDecodeError /home*/* stopped e89c80d823b9400baf869a4f8b807ca5 1574154126 6304470 6304470 0
1573549325 None /home*/* stopped 72f7959011234292be5d45f7e702b390 1573549325 5986529 5986529 8
1572944526 None /home*/* stopped 9a317b6d873440649f5c5ce55a113ce8 1572944526 5668026 5668026 5
1572336127 None /home*/* stopped 22769cc729424e559bf4400119823300 1572336127 5613829 5613829 2
1571731326 None /home*/* stopped 7297d7b796d34d0c8efbf8fde4ea4b89 1571731326 5573682 5573682 1
1571126525 None /home*/* stopped 80f62f757f09406286a46ea994ad87fd 1571126525 5553637 5553637 0
1570521725 None /home*/* stopped 52d7652acfe84460aeefc6a16cc6ab68 1570521725 5515505 5515505 0
1569916926 None /home*/* stopped b007d09797b24da3a093e881132cb56a 1569916926 5495460 5495460 124
1568298028 None /home*/* stopped 3d5c486865d34be7a15a75eacb9b66f8 1568298028 5761169 5761169 332


Perhaps we're hitting a resource limit with intensity_ram?


$ /usr/bin/imunify360-agent malware on-demand status
created: 1582146418
exclude_patterns:
- /home*/*/mail/*
- /home*/*/.trash/*
- /home*/*/lscache/*
file_patterns: null
follow_symlinks: false
intensity_cpu: 1
intensity_io: 1
intensity_ram: 1024
path: /home*
phase: preparing file list
progress: 0
queued: 0
scan_type: on-demand
scanid: 5cc91e7152104edeb53fbc4e52877ad6
started: 1582146418
status: running


I see there is an "--intensity-ram" argument reported by help:


usage: imunify360-agent malware on-demand start [-h] [--file-mask FILE_MASK]
[--intensity {low,moderate,high}]
[--intensity-io INTENSITY_IO]
[--intensity-ram INTENSITY_RAM]
[--follow-symlinks | --no-follow-symlinks]
--path PATH
[--intensity-cpu INTENSITY_CPU]
[--ignore-mask IGNORE_MASK]
[--json] [--verbose]

optional arguments:
-h, --help show this help message and exit
--file-mask FILE_MASK
--intensity {low,moderate,high}
--intensity-io INTENSITY_IO
--intensity-ram INTENSITY_RAM
--follow-symlinks
--no-follow-symlinks
--path PATH
--intensity-cpu INTENSITY_CPU
--ignore-mask IGNORE_MASK
--json return data in JSON format
--verbose, -v


However "--intensity-ram" is not mentioned in the documentation, which only lists "--intensity-cpu" and "--intensity-io" as possible intensity flags. It doesn't appear in the GUI settings in the WHM plugin either.

If at all possible, we don't want a malware scan to consume multiple GB of RAM, but we also don't want an entire scan to fail with files left unscanned because the RAM limit is reached, either.

We can work around this in a few ways if needed, but the behavior seems like a bug. Additionally I'm not entirely comfortable with using undocumented features in our automation. At present there seems to be a big mismatch between features documented at docs.imunify360.com and feature documented with the CLI help. The CLI help seems more complete, but it's missing details on some features which are in the documentation. As a result it is taking guesswork and experimentation for us to learn how the software actually works.
  1. 20.02.2020 15:02:47
  2. # 3
Sergey Khristich Accepted Answer
Posts: 433
Joined: 20.05.2019
0
Votes
Undo
Hello John,
Thank you for the information provided.
You are right "intensity_ram" is really undocumented.
To avoid a similar problem, you can raise it in the configuration or scan smaller volumes.
If you have any other questions, feel free to ask here. Thank you for contacting us.
Marketing Manager
  1. 20.02.2020 18:02:51
  2. # 4
John Carpenter Accepted Answer
Posts: 6
Joined: 28.06.2019
0
Votes
Undo
Thank you for your response.

Is memory usage expected to grow depending on the amount of files or directories being scanned? I notice the scanner seems to walk directories and build a directory list before scanning files. Does it need to hold all target directories in memory during the scan?

In other words is it a bad idea to try to scan a large complex filesystem in a single scan operation?
  1. 21.02.2020 09:02:01
  2. # 5
Sergey Khristich Accepted Answer
Posts: 433
Joined: 20.05.2019
0
Votes
Undo
Hello John,
It doesn't hold all target directories in RAM. The scanner needs memory to build a report. But yes, I recommend splitting the scan into smaller ones. Let me know if you have any questions. Thanks.
Marketing Manager
  1. 21.02.2020 20:02:09
  2. # 6
John Carpenter Accepted Answer
Posts: 6
Joined: 28.06.2019
0
Votes
Undo
Thank you Sergey for the additional information.

The problem that I'm trying to solve, is that I want to do automated periodic scans of all hosting accounts, while ignoring certain wildcard directories. For example I don't want to scan mail, or the cPanel File Manager's .trash directory.

Here are the methods I've tried:

* The background scanner almost does what I need, but it does not seem to allow for wildcard ignore masks.

* "malware user scan" will let me queue a scan for each user and almost does what I need, except again it does not accept "--ignore-mask" or "--intensity*" options.

* "malware on-demand start" almost does what I need. It accepts "--ignore-mask" and "--intensity" arguments. But it does not allow me to queue scans. It fails if a scan is already in progress.

* Experimenting with CLI -h flag, I just found "malware on demand queue put" which is an undocumented command. It appears nowhere in the documentation, and it's not even listed as an "Available command" according to CLI help:


$ /usr/bin/imunify360-agent malware on-demand -h
usage: imunify360-agent malware on-demand [-h]
{list,queue,start,status,stop} ...

positional arguments:
{list,queue,start,status,stop}
Available commands
list
start
status
stop

optional arguments:
-h, --help show this help message and exit
$ /usr/bin/imunify360-agent malware on-demand queue -h
usage: imunify360-agent malware on-demand queue [-h] {put,remove} ...

positional arguments:
{put,remove} Available commands
put
remove

optional arguments:
-h, --help show this help message and exit


This looks like it will do what I need, queue a scan for a given directory with ignore masks and intensity options. Am I understanding this correctly?

Your product is very good overall, but the documentation and help output are in serious need of improvement IMHO. My use case should be very straightforward, but it's taking a lot of experimentation and trial and error to accomplish it using undocumented features. Furthermore I am concerned about using undocumented features in our scripts. With behavior not specified in documentation, I am never sure if I understand the intended functionality, and I'm concerned it might change in future versions after scripting is deployed. Do you understand my concerns?
  1. 21.02.2020 22:02:04
  2. # 7
Sergey Khristich Accepted Answer
Posts: 433
Joined: 20.05.2019
0
Votes
Undo
Hello John,
Thanks for continuing.

The questions arose because it seems that an on-demand scan started as follows, is not scanning every home directory for reasons unknown:

# imunify360-agent malware on-demand start --path '/home*/*' --no-follow-symlinks --intensity low


Can you open a support ticket https://cloudlinux.zendesk.com/hc/en-us/requests/new so we can take a closer look at your system? You can post the ticket number here and we'll link this thread to it.


Your product is very good overall, but the documentation and help output are in serious need of improvement IMHO. My use case should be very straightforward, but it's taking a lot of experimentation and trial and error to accomplish it using undocumented features. Furthermore I am concerned about using undocumented features in our scripts. With behavior not specified in documentation, I am never sure if I understand the intended functionality, and I'm concerned it might change in future versions after scripting is deployed. Do you understand my concerns?

We really appreciate your opinion and wishes. In addition, we are currently preparing documentation updates and there will be a lot of things added there, but some of the commands that are official and will not be displayed in the documents.
Drop me a line if I can do anything else for you. Thank you for contacting us.
Marketing Manager
  • Page :
  • 1


There are no replies made for this post yet.
Be one of the first to reply to this post!
Greg
Submit Your Response
Upload files or images for this discussion by clicking on the upload button below. Supports gif,jpg,png,zip,rar,pdf
• Insert • Remove Upload Files (Maximum File Size: 2 MB)
Captcha
To protect the site from bots and unauthorized scripts, we require that you enter the captcha codes below before posting your question.