Hi there! I'd really like to work on this. I wrote a similar utility for mapping infrastructure on aws at my last job but never really got to dig into it. I used d3 for the visualization and grabbed the map in json. The cron job could generate a webpage or data for some little app (we'd use flask for this mebbe at my last job). It would be a useful service all around. I want it in my day job, too.

Replying to [comment:2 munjeli]:

Hi there! I'd really like to work on this. I wrote a similar utility for mapping infrastructure on aws at my last job but never really got to dig into it. I used d3 for the visualization and grabbed the map in json. The cron job could generate a webpage or data for some little app (we'd use flask for this mebbe at my last job). It would be a useful service all around. I want it in my day job, too.

Got a bit busy myself so have at it.

Yay! I pulled in lousab, we'll get on it. Thanks for giving us something real to do right off the bat. Also this is a nice task to explore the infrastructure. We'll make the most of it!

cron script file updated
infrastructure.sh

Hi i'm coming back. Starting from sh marc84 script file i'd like to generate an html page with YAML python following the same as https://apps.fedoraproject.org/

I'm working on it.

Are folks still interested in this? Any progress?

@lousab @munjeli @linuxmodder @marc84

Hmmm that's a good question. I might be? I got really busy there for a while, but I'm working less these days and still want this thing. Let me look for the data first and then I'll make the call.

Honestly forgot this was still active and @kevin I am still interested in helping with it, will try to turn out some cycles before the weekend on this.

so, thats a flask app? I think we wanted something more to generate a static map than an app... I'm not sure how best to display such a thing tho. We do have a nagios map, but it's... gigantic. Perhaps any map would be.

https://nagios.fedoraproject.org/nagios//map.php?host=all (you need to have done a kinit yourusername@FEDORAPROJECT.ORG and have a kerberos to access) is the nagios map.

It's pretty difficult to see much there.

so, thats a flask app? I think we wanted something more to generate a static map than an app... I'm not sure how best to display such a thing tho. We do have a nagios map, but it's... gigantic. Perhaps any map would be.

https://nagios.fedoraproject.org/nagios//map.php?host=all (you need to have done a kinit yourusername@FEDORAPROJECT.ORG and have a kerberos to access) is the nagios map.

It's pretty difficult to see much there.

I have no idea where we are here... if folks want to move this forward please do.

Would it be possible to take the data from Nagios regarding online (connected/down) servers of ours and do something like pipe that information into a separate application?

If this is possible, then couldn't I just write a program to write this information to a log file and regularly run the program using cron? that way, we would just have to grep it's contents.

@kevin @smooge - I've done just a bit of digging on this one in the Nagios docs. Has anyone considered any of the following?

1. Creating a page of different views available via Nagios?

For example, this is just a custom URL with a different layout, with virthost11 set as root;

https://nagios.fedoraproject.org/nagios//map.php?root=virthost11.phx2.fedoraproject.org&layout=9

Layouts and custom URL info is on this nagios page.

1. NagVis - Havent looked at it much, but it is OS/GPLv2

2. Regarding the original request, is Ansible Tower the answer here?

@mizdebsk did you have any ideas on how this should be visualized?

I have some ideas, but I'm super busy right now and I don't have time for anything except critical work. I will provide more details in second half of August when I'm back from vacation.

@mizdebsk thats perfect! We decided to do an apprentice workday on this item sometime after flock-ish. I'll put out an email to the infra list.

Here's a rough version of the json output of my data mining script we discussed on last week's meeting. I'll keep digging for more useful data.

It looks like most (possibly all) of the information needed to generate this is available via Nagios CGI queries to its JSON generators. There is a simple HTML form to play with the queries at https://nagios.fedoraproject.org/nagios/jsonquery.html. Instead of a script to generate an HTML map, maybe we just need a JavaScript heavy HTML page to pull and parse data from the Nagios CGI end points?

I am removing easyfix from this. It looks simple but we keep running into 'oh wow that is going to take a lot of time' on it.

This ticket has been open for 4 years and it turned out to not be such an easy thing to do.

Do we still think it's needed?

I have just started to write a quick script that works based on ks3's comment, but it seems that we're just reinventing the wheel: there are some nagios plugins that would fit.
What about http://www.nagvis.org ?

Hi @mattia

Thank you and the other apprentices for doing this research. In looking at this over different ways, I think we are approaching this backwards with the 'map' being the icing on the cake which needs a lot of layers baked first.

The nagios data is not going to really tell where a system is or what its relationship is with other items unless that is recorded somewhere. That data would need to be kept up to date in the ansible playbooks.

Currently nagios is mainly a bag of checks without any knowledge of where things are, what they do with each other or how one item relies on the other. How to do this is needed to be sorted and solved before we can 'map' out where things are.

What needs to be done is a better scoped out engineered monitoring system AND collecting more data into each ansible hosts info for setup and creation so that this can be populated. Doing the survey of what information is wanted, what is wanted to be viewed and how to collect that data would help getting some of the layers baked so that any icing can be done.

I think it would be best to close this and start specing out the solution in steps from the bottom up. The maps are important but if they can't populated from current infra, then it is work that can't be implemented.