Zabbix JSONpath LLD

3 Low-level discovery

Overview

Low-level discovery provides a way to automatically create items, triggers, and graphs for different entities on a computer. For instance, Zabbix can automatically start monitoring file systems or network interfaces on your machine, without the need to create items for each file system or network interface manually. Additionally, it is possible to configure Zabbix to remove unneeded entities automatically based on actual results of periodically performed discovery.

These macros are used in names, keys and other prototype fields where they are then substituted with the received values for creating real items, triggers, graphs or even hosts for each discovered entity. See the full list of options
for using LLD macros.

Note that since Zabbix 4.2
, the format of the JSON returned by low-level discovery rules has been changed. It is no longer expected that the JSON will contain the “data” object. Low-level discovery will now accept a normal JSON containing an array, in order to support new features such as the item value preprocessing and custom paths to low-level discovery macro values in a JSON document.

As a result of the changes above, newer agents no longer will be able to work with an older Zabbix server.

See also: Discovered entities

Configuring low-level discovery

We will illustrate low-level discovery based on an example of file system discovery.

• Click on Discovery
  in the row of an appropriate template/host

• Click on Create discovery rule
  in the upper right corner of the screen

• Fill in the discovery rule form with the required details

Discovery rule

The discovery rule form contains five tabs, representing, from left to right, the data flow during discovery:

• Discovery rule
  — specifies, most importantly, the built-in item or custom script to retrieve discovery data

• Preprocessing
  — applies some preprocessing to the discovered data

• LLD macros
  — allows to extract some macro values to use in discovered items, triggers, etc

• Filters
  — allows to filter the discovered values

• Overrides
  — allows to modify items, triggers, graphs or host prototypes when applying to specific discovered objects

The Discovery rule
tab contains the item key to use for discovery (as well as some general discovery rule attributes):

All mandatory input fields are marked with a red asterisk.

Discovery rule history is not preserved.

Preprocessing

The Preprocessing
tab allows to define transformation rules to apply to the result of discovery. One or several transformations are possible in this step. Transformations are executed in the order in which they are defined. All preprocessing is done by Zabbix server.

Note that if the discovery rule has been applied to the host via template then the content of this tab is read-only.

Custom macros

The LLD macros
tab allows to specify custom low-level discovery macros.

Custom macros are useful in cases when the returned JSON does not have the required macros already defined. So, for example:

The extracted values can be used in discovered items, triggers, etc. Note that values will be extracted from the result of discovery and any preprocessing steps so far.

Filter

The Filters
tab contains discovery rule filter definitions allowing to filter discovery values:

A mistake or typo in the regular expression used in LLD rule may cause deleting thousands of configuration elements, historical values and events for many hosts. For example, an incorrect “File systems for discovery” regular expression may cause deleting thousands of items, triggers, historical values and events.

Zabbix database in MySQL must be created as case-sensitive if file system names that differ only by case are to be discovered correctly.

Override

The Override
tab allows to set rules to modify the list of item, trigger, graph and host prototypes or their attributes for discovered objects that meet given criteria.

All mandatory parameters are marked with red asterisks.

Configuring an operation

Form buttons

Buttons at the bottom of the form allow to perform several operations.

Item prototypes

Context-specific escaping of low-level discovery macros is performed for safe use in regular expression and XPath preprocessing parameters.

Attributes that are specific for item prototypes:

We can create several item prototypes for each file system metric we are interested in:

Mass update

option is available if you want to update properties of several item prototypes at once.

Trigger prototypes

We create trigger prototypes in a similar way as item prototypes:

Attributes that are specific for trigger prototypes:

You can define dependencies
between trigger prototypes as well (supported since Zabbix 3.0). To do that, go to the Dependencies
tab. A trigger prototype may depend on another trigger prototype from the same low-level discovery (LLD) rule or on a regular trigger. A trigger prototype may not depend on a trigger prototype from a different LLD rule or on a trigger created from trigger prototype. Host trigger prototype cannot depend on a trigger from a template.

Graph prototypes

We can create graph prototypes, too:

Attributes that are specific for trigger prototypes:

Finally, we have created a discovery rule that looks as shown below. It has five item prototypes, two trigger prototypes, and one graph prototype.

Note
: For configuring host prototypes, see the section about host prototype
configuration in virtual machine monitoring.

Discovered entities

The screenshots below illustrate how discovered items, triggers, and graphs look like in the host’s configuration. Discovered entities are prefixed with an orange link to a discovery rule they come from.

Note that discovered entities will not be created in case there are already existing entities with the same uniqueness criteria, for example, an item with the same key or graph with the same name. An error message is displayed in this case in the frontend that the low-level discovery rule could not create certain entities. The discovery rule itself, however, will not turn unsupported because some entity could not be created and had to be skipped. The discovery rule will go on creating/updating other entities.

Items (similarly, triggers and graphs) created by a low-level discovery rule will be deleted automatically if a discovered entity (file system, interface, etc) stops being discovered (or does not pass the filter anymore). In this case the items, triggers and graphs will be deleted after the days defined in the Keep lost resources period
field pass.

When discovered entities become ‘Not discovered anymore’, a lifetime indicator is displayed in the item list. Move your mouse pointer over it and a message will be displayed indicating how many days are left until the item is deleted.

If entities were marked for deletion, but were not deleted at the expected time (disabled discovery rule or item host), they will be deleted the next time the discovery rule is processed.

Entities containing other entities, which are marked for deletion, will not update if changed on the discovery rule level. For example, LLD-based triggers will not update if they contain items that are marked for deletion.

Other types of discovery

Data limits for return values

If data has to go through Zabbix proxy it has to store this data in database so database limits
apply.

Multiple LLD rules for same item

Since Zabbix agent version 3.2 it is possible to define several low-level discovery rules with the same discovery item.

Creating custom LLD rules

It is also possible to create a completely custom LLD rule, discovering any type of entities — for example, databases on a database server.

To do so, a custom item should be created that returns JSON, specifying found objects and optionally — some properties of them. The amount of macros per entity is not limited — while the built-in discovery rules return either one or two macros (for example, two for filesystem discovery), it is possible to return more.

Allowed symbols for LLD macro names are 0-9
, A-Z
, _
, .

Lowercase letters are not supported in the names.

You don’t have to use macro names FSNAME/FSTYPE with custom LLD rules, you are free to use whatever names you like. In case JSONPath is used then LLD row will be an array element that can be an object, but it can be also another array or a value.

Using LLD macros in user macro contexts

 // remove duplicates
var lookup = {};
var items = JSON.parse(value);
var result = [];
for (var item, i = 0; item = items[i++];) { var url = item.url; if (!(url in lookup)) { var row = {}; lookup[url] = 1; row["url"] = item.url; row["price"] = item.price; row["District"] = item.District; row["City"] = item.City; row["Street"] = item.Street; row["Rooms"] = item.Rooms; row["Area"] = item.Area; row["Floor"] = item.Floor; row["Series"] = item.Series; row["Type"] = item.Type; row["Facilities"] = item.Facilities; result.push(row); }
}
return JSON.stringify(result); 

match real ip addresses in the hosts file:

 return value
.match(/^[^#]+/gm)
.sort().join("\n")
.match(/^[1-9]+.*$/gm)
.sort().join("\n")
.trim()
.replace(/^\s*[\r\n]/gm, ''); 

work with run extra scenario of a keyword is found

 // look for keyword "pod" (including double quotes)
if (value.search("\"pod\"") > 0) {
// if pod has been found then execute extra JSONPath to find name
// Set the plain text to be a JSON object
var jsonData = JSON.parse(value);
// Count length of array. How many pods has been failed
lenghtOfArray=jsonData.length;
// Define an array to put inside all failed pods
var answerArray = [];
// Locate each pod and put inside answer Array
for(var pod = 0; pod < lenghtOfArray; pod++){
answerArray.push(jsonData[pod].metric.pod);
}
// Return all failed pods, and convert array to text with a comma as dividor
return answerArray.join(", ");
}
else {
// If "pod" was not found, then nothing happens
return 0
} 

Sort lines alphabetically:

 var data='beta\nalpha\ndelta';
var array=data.match(/.*/gm);
return array.sort().join("\n"); 

Remove comments and sort alphabetically:

 return value
.match(/^[^#]+/gm)
.sort().join("\n")
.trim()
.replace(/^\s*[\r\n]/gm, '')
.match(/.*/gm).sort()
.join("\n")
.replace(/^\s*[\r\n]/gm, ''); 

 var lld = [];
// this line is important to match all required patterns right away. Otherwise the upcoming code will fail.
var lines = value.match(/^.*:\d+.*\"[^\"]+\".*$/gm);
// go through all lines which match critera
var lines_num = lines.length;
for (i = 0; i < lines_num; i++)
{ var row = {}; row["port"] = lines[i].toString().match(/:([0-9]+)/)[1]; row["process"] = lines[i].toString().match(/\"([^\"]+)\"/)[1]; lld.push(row);
} return JSON.stringify(lld);  

 var arr1 = [ 1, 2, 3, 5 ];
var arr2 = [ 1, 4, 5, 6 ];
// merge, remove dublicates, sort
var arr3 = arr1.concat(arr2)
.reduce(function(a,b){if(a.indexOf(b) < 0)a.push(b);return a;},[])
.sort(function(a, b){return a - b});
return arr3; 

Count of lines:

 return value.split(/\r\n|\r|\n/).length; 

Specific services to monitor:

 var lines = '{$SERVICES_TO_MONITOR}'.split("|");
var lld = [];
var lines_num = lines.length;
for (i = 0; i < lines_num; i++)
{ var row = {}; row["{#NAME}"] = lines[i] lld.push(row);
}
return JSON.stringify(lld); 

 service1|service2 

Sort a floating number array from biggest to smalles:

 return value.match(/[0-9\.]+/gm).sort(function(a, b){return a - b}).reverse(); 

value mod 180:

 return value % 180 

Occurence of pattern:

 var temp = "Wel to come to W3Docs to asd";
return (temp.match(/to/g) || []).length; 

Validate and extract:

 var lld = [];
if (value.match(/1 packets transmitted, 1 received/)) {
var lines = value
.match(/^.*1 packets transmitted, 1 received/gm)
.join("\n")
.replace(/^PING /gm,"")
.match(/^[0-9\.]+/gm);
var lines_num = lines.length;
for (i = 0; i < lines_num; i++)
{ var row = {}; row["{#IP.POS}"] = lines[i] lld.push(row);
}
return JSON.stringify(lld);
}
else
{
return JSON.stringify(lld);
} 

Remove leading spaces:

 .replace(/^\s+/gm,''); 

Remove leading and trailing spaces:

 .replace(/^\s+|\s+$/gm,'') 

 var string="kaste ta 13 da suns 66 iet"
var re=/\d+/g;
return string.match(re) ? re.exec(string) : 0; 

Extract all numbers:

 return "This 13 this is another number 54 and third number 222".match(/\d+/g); 

Detect if value is there with regex

 return value.match(/.*/) ? value : 0; 

Detect if value is there with strings

 if (value.search("kaste") > 0) {
return value;
}
else {
return 0
} 

Extract first group:

 var regexp = /<ServiceName xsi:type="xsd:string">([\w\s]+)<\/ServiceName>/gm;
var match = regexp.exec(value);
return match.length; 

Escape special characters:

 var jsonData = JSON.parse(value);
lenghtOfArray=jsonData.length;
for(var row = 0; row < lenghtOfArray; row++){
jsonData[row]. Pattern = jsonData[row]. Pattern
.replace(/[-[\]{}()*+?,\\^$|#\s]/g, '\\
 var jsonData = JSON.parse(value);
lenghtOfArray=jsonData.length;
for(var row = 0; row < lenghtOfArray; row++){
jsonData[row].Pattern = jsonData[row].Pattern
.replace(/[-[\]{}()*+?.,\\^$|#\s]/g, '\\var jsonData = JSON.parse(value);
lenghtOfArray=jsonData.length;
for(var row = 0; row < lenghtOfArray; row++){
jsonData[row].Pattern = jsonData[row].Pattern
.replace(/[-[\]{}()*+?.,\\^$|#\s]/g, '\\(var jsonData = JSON.parse(value);
lenghtOfArray=jsonData.length;
for(var row = 0; row < lenghtOfArray; row++){
jsonData[row].Pattern = jsonData[row].Pattern
.replace(/[-[\]{}()*+?.,\\^$|#\s]/g, '\\
0amp;');
}
return JSON.stringify(jsonData);)amp;');
}
return JSON.stringify(jsonData);amp;');
}
return JSON.stringify(jsonData); 
amp;');
 }
return JSON.stringify(jsonData);
Remove LLD macro after ODBC discovery:
 return value
.replace(/{#ID}/gm,'ID')
.replace(/{#NAME}/gm,'NAME')
.replace(/{#VALUE}/gm,'VALUE')
.replace(/{#CREATED_ON}/gm,'CREATED_ON')
.replace(/{#UPDATED_ON}/gm,'UPDATED_ON')
.replace(/{#ISDELETED}/gm,'ISDELETED'); 
Sort and integer arrar:
 return value.sort(function(a, b){return a - b}); 
   
Remove empy lines:        
Print new ports from context: 
        
Transfor SNMP discovery:
JSONPath:     

Preprocessing to transform SNMPINDEX, erase everything to the right: 
        
Dirs and sizes:     

Summ all integers inside text '111+222+333': 
            
  
Check a pattern in one input:         
Remove all new line characters:   
  
Remove all spaces:         
Delete all double quotes:   
  
Insert at the start of line:         
Insert at the end of line:   
  
Delete last character at the end:              
        

Create a discovery rule from content 
. Solution: 
 return value.replace(/,/g,"\"},{\"{#LANG}\":\"").replace(/^/,"{\"data\":[{\"{#LANG}\":\"").replace(/$/,"\"}]}") 
 Search for links:
 return value.match(/(http|https|ftp|ftps)\:\/\/[a-zA-Z0-9\-\.\/_?&=]+/gm).reduce(function(a,b){if(a.indexOf(b)< 0)a.push(b);return a;},[]).join("\n") 
Format /etc/passwd for discovery:
 return value.match(/^.*bash$/gm).join("\n").replace(/..bin.bash$/gm,"\"},").replace(/:.*:/gm,"\",\"{#PATH}\":\"").replace(/^/gm,"{\"{#USER}\":\"").replace(/^/,"{\"data\":[").replace(/.$/,"]}") 
 https:\/\/catonrug.blogspot.com\/2020\/01\/find-files-in-specific-date-time-between.html 
 https://catonrug.blogspot.com/2020/01/find-files-in-specific-date-time-between.html 
 return value.replace(/https?:\\\/\\\/([a-z\.]+)\\\/([0-9]+)\\\/([0-9]+)\\\//gm, "https:\/\/$1\/$2\/$3\/"); 
 var lld = [];
var lines = value.match(/(http|https|ftp|ftps)\:\/\/[a-zA-Z0-9\-\.\/_?&=]+/gm).reduce(function(a,b){if(a.indexOf(b)< 0)a.push(b);return a;},[]);
var lines_num = lines.length;
for (i = 0; i < lines_num; i++)
{ var row = {}; row["{#URL}"] = lines[i] lld.push(row);
}
return JSON.stringify(lld); 
Lowercase whole array:
 return JSON.parse(JSON.stringify(value, function(a, b) { return typeof b === "string" ? b.toLowerCase() : b })); 
HEX to ASCII:
 var out = "";
for (var i = 0, nums = value.split(" "); i < nums.length; i++) { out += String.fromCharCode(parseInt(nums[i], 16));
}
return out; 
 var obj = JSON.parse(value), result = obj.filter(function(e) { return e['{#' + 'TABLESPACE}'] == '{#TABLESPACE}'; });
return result[0]['{#' + 'TOTAL_MB}'] 
Replace bash variables:
 var lld = [];
// check if there are any links at all
if (value.match(/(http|https|ftp|ftps)\:\/\/[a-zA-Z0-9\-\.\/\$_?&=]+/gm)) {	// if there are links, then extract and make sure there are no dublicates:	var lines = value.match(/(http|https|ftp|ftps)\:\/\/[a-zA-Z0-9\-\.\/\$_?&=]+/gm).reduce(function(a,b){if(a.indexOf(b)< 0)a.push(b);return a;},[]);	var lines_num = lines.length;	for (i = 0; i < lines_num; i++)	{	var row = {};	// a patch to extract a variable (the one with a dollar sign)	// if the following input turn out some sort of bash code	if (lines[i].match(/\$[a-z]+/gm)) {	// extract full title of variable and value	var variable = value.match(/([a-z]+)=([a-z0-9\.]+)/gm);	// how many variables are in bash code	var count_of_variables = variable.length;	// go through all variables in bash code	for (j = 0; j < count_of_variables; j++) {	var regexp = /([a-z]+)=([a-z0-9\.]+)/gm;	var unpack = regexp.exec(variable[j]);	var var_name = ' 
   
 
 For example when you changed the settings
Just disable, wait a bit and enable again. 
 
 Run playbook on single host 
 
 ansible_playbook -l somehost somplay.yml 
 Find hosts with hostmacro defined
 select h.host, m.macro, m.value from hosts h, hostmacro m where macro like '%FOO%' and h.hostid = m.hostid; 
 Unsupported item key
 This might mean it's expecting a value from the script you're calling.
 echo 1 
 remember: not supported is not disabled, server/proxy will try again after interval
  became not supported: Not supported by Zabbix Agent 
 No active checks on server
 Hostname in agent config (-kagent.hostname) must match name on server
 simple no connection possible? firewall?
 First network error
 Setting  Timeout 
 in server configuration
 also Timeout in agents?
  
 This probably means the agent returned 1\n1 
 Install recent zabbix on CentOS/RHEL
 rpm -ivh  https://repo.zabbix.com/zabbix/3.4/rhel/7/x86_64/zabbix-release-3.4-2.el7.noarch.rpm 
yum install zabbix-agent 
 LLD with JSON
 if you want multiple keys, use jsonpath like
 $[?(@.share=='{#FSTYPE}' && @.name=='{#NAME}')].size.first() 
  testing jsonpath preprocessing 
 Zabbix escalator processes more than 75% busy
 probably high system load overall
 Zabbix agent paths
 /etc/zabbix/zabbix_agentd.conf.d/
/etc/zabbix/zabbix_agentd.conf 
 Show cpu utilization
  fuzzytime on command line 
 TS=lotsofseconds 
 output in hours
 echo $(( ($(date +%s) - $TS) / 3600 )) 
  
 Could be something /tmp related
 
  Zabbix_get returns nothing 
  best look at log on agent side
 
 
 
 Check table 'ids'
  Incorrect trigger expression. Host "xx" does not exist or you have no access to this host.
 Means there's no related item.
 Zabbix agent active
 Have port 10051 open and:
 ActiveServer zabbix.ser.ver 
 Set Agent IP to 0.0.0.0
 Template App MySQL
 create user 'monitor'@'localhost' identified by auth_socket;
grant PROCESS,SHOW DATABASES,SHOW VIEW on *.* to 'monitor'@'localhost';
flush privileges; 
 The information displayed may not be current
 Might be selinux:  http://sysads.co.uk/2013/11/zabbix-server-running-alert/ 
 Retrying in 10 seconds
 try upping max_connections
 Agent side ping check
 UserParameter=pingtime[*],fping -e $1|sed 's/^.*(\([0-9].*\) ms).*$/\1/g'
UserParameter=pingalive[*],fping $1|grep -q alive;echo $? 
 Installing Zabbix from git
 git clone  https://github.com/zabbix/zabbix.git 
cd zabbix
./bootstrap.sh
./configure --help
autoreconf -fvi 
 If you don't have a Makefile, try
 ./config.status Makefile 
 ./configure 
 Most frequent items in history_uint
 select itemid,count(itemid) as freq from history_uint group by itemid order by freq desc limit 5; 
 select name from items where itemid = whateveryoufind; 
 Discovery data example
 Output of a discovery script should look like:
 {"data":[ {"{#VAR1}":"value11","#{VAR2":"value12"}, {"{#VAR1}":"value21","#{VAR2":"value22"}
]} 
  
  Discover: value must be a JSON object 
 Could mean you need to escape slashes, check output with zabbix_get
  Cannot create item: item with the same key 
 Cannot send list of active checks
 /usr/sbin/zabbix_agentd -t 'agent.hostname' 
 If in agent log: most likely ServerActive is defined in agent config, while not used at all
 It is also possible agent is sending some active check to server while host is monitored via proxy.
 In proxy/server log:
 most likely  Hostname 
 in agent config does not match hostname used on server.
 + unpack[1];
 var var_value = unpack[2];
 // do replacement for the link
 for (k = 0; k < (lines[i].match(/\$/g) || []).length; k++) {
 lines[i] = lines[i].replace(var_name,var_value);
 }
 }
 row["{#POST. URL}"] = lines[i];
 } else {
 row["{#POST. URL}"] = lines[i];
 }
 lld.push(row);
 }
 return JSON.stringify(lld);
} else {
 return JSON.stringify(lld);
 }
For example when you changed the settings
 Just disable, wait a bit and enable again.
Run playbook on single host
ansible_playbook -l somehost somplay.yml
Find hosts with hostmacro defined
select h.host, m.macro, m.value from hosts h, hostmacro m where macro like '%FOO%' and h.hostid = m.hostid;
Unsupported item key
This might mean it's expecting a value from the script you're calling.
echo 1
remember: not supported is not disabled, server/proxy will try again after interval
became not supported: Not supported by Zabbix Agent
No active checks on server
Hostname in agent config (-kagent.hostname) must match name on server
simple no connection possible? firewall?
First network error
Setting Timeout in server configuration
also Timeout in agents?
This probably means the agent returned 1\n1
Install recent zabbix on CentOS/RHEL
rpm -ivh https://repo.zabbix.com/zabbix/3.4/rhel/7/x86_64/zabbix-release-3.4-2.el7.noarch.rpmyum install zabbix-agent
LLD with JSON
if you want multiple keys, use jsonpath like
$[?(@.share=='{#FSTYPE}' && @.name=='{#NAME}')].size.first()
testing jsonpath preprocessing
Zabbix escalator processes more than 75% busy
probably high system load overall
Zabbix agent paths
/etc/zabbix/zabbix_agentd.conf.d/
/etc/zabbix/zabbix_agentd.conf
Show cpu utilization
fuzzytime on command line
TS=lotsofseconds
output in hours
echo $(( ($(date +%s) - $TS) / 3600 ))
Could be something /tmp related
Zabbix_get returns nothing
best look at log on agent side
Check table 'ids'
Incorrect trigger expression. Host "xx" does not exist or you have no access to this host.
Means there's no related item.
Zabbix agent active
Have port 10051 open and:
ActiveServer zabbix.ser.ver
Set Agent IP to 0.0.0.0
Template App MySQL
create user 'monitor'@'localhost' identified by auth_socket;
grant PROCESS,SHOW DATABASES,SHOW VIEW on *.* to 'monitor'@'localhost';
flush privileges;
The information displayed may not be current
Might be selinux: http://sysads.co.uk/2013/11/zabbix-server-running-alert/
Retrying in 10 seconds
try upping max_connections
Agent side ping check
UserParameter=pingtime[*],fping -e $1|sed 's/^.*(\([0-9].*\) ms).*$/\1/g'
UserParameter=pingalive[*],fping $1|grep -q alive;echo $?
Installing Zabbix from git
git clone https://github.com/zabbix/zabbix.gitcd zabbix
./bootstrap.sh
./configure --help
autoreconf -fvi
If you don't have a Makefile, try
./config.status Makefile
./configure
Most frequent items in history_uint
select itemid,count(itemid) as freq from history_uint group by itemid order by freq desc limit 5;
select name from items where itemid = whateveryoufind;
Discovery data example
Output of a discovery script should look like:
{"data":[ {"{#VAR1}":"value11","#{VAR2":"value12"}, {"{#VAR1}":"value21","#{VAR2":"value22"}
]}
Discover: value must be a JSON object
Could mean you need to escape slashes, check output with zabbix_get
Cannot create item: item with the same key
Cannot send list of active checks
/usr/sbin/zabbix_agentd -t 'agent.hostname'
If in agent log: most likely ServerActive is defined in agent config, while not used at all
It is also possible agent is sending some active check to server while host is monitored via proxy.
In proxy/server log:
 most likely Hostname in agent config does not match hostname used on server.
 </p