Can you explain a bit more about your needs here? "a server" is pretty vuage. ;)

What is the site written in (some framework?), can it emit static content to be served from our proxies?

Can it be cached behind our proxies and have a single backend? or pair of backends?

What os/version does it run on?

Would a cloud instance work for it?

What expectations of uptime does it have?

This site will be completely static, e.g. HTML, CSS, JavaScript for search. It's generated by Jekyll, but we don't need the build process to happen on the server.

What os/version does it run on?

Does not matter.

Would a cloud instance work for it?

Yes, certainly.

Replying to [comment:1 kevin]:

Can you explain a bit more about your needs here? "a server" is pretty vuage. ;)
Yes, it is, excuse me :-)

What is the site written in (some framework?), can it emit static content to be served from our proxies?
As jstribny said, it will be a static website. There would be also a script (in python or ruby) that would refresh the homepage, let's say, every 15 minutes with fresh blog posts from an rss channel.

Can it be cached behind our proxies and have a single backend? or pair of backends?
Yes.

What expectations of uptime does it have?
I would say the same as sites such as getfedora.org or the Fedora Magazine.

ok, thanks for the info. :)

A few more questions:

Is the toolchain/things you need to build the site all packaged in epel? in fedora?

It sounds to me like the ideal way might be a backend server that just builds the site and syncs to our proxies, then the proxies can serve the static content with httpd. That would make it close to users and faster.

Is the toolchain/things you need to build the site all packaged in epel? in fedora?

It sounds to me like the ideal way might be a backend server that just builds the site and syncs to our proxies, then the proxies can serve the static content with httpd. That would make it close to users and faster.

That would be the best. The site is however built using Jekyll that is not yet part of Fedora. Get Jekyll there would mean package ~20 new packages so it's not possible to do now in our time constrain although we can head that way towards the feature. I could most likely package it far sooner in Copr if that would be acceptable.

If not, we can upload just the static site, and have a separate script just for refreshing the blogs. For that a simple Python/Ruby runtime could be installed from base Fedora.

For dev and stg just a copr or the infra repo should work, but for production we really prefer being in epel.

So, really this is a RFR (Request For Resources).

https://fedoraproject.org/wiki/Request_For_Resources

For the first part (a dev instance), I can make you a cloud instance for that. Or do you want to just skip that since you have an existing dev instance somewhere else? There still might be some use in testing copr builds and such...

Replying to [comment:6 kevin]:

For dev and stg just a copr or the infra repo should work, but for production we really prefer being in epel.

So, really this is a RFR (Request For Resources).

https://fedoraproject.org/wiki/Request_For_Resources

For the first part (a dev instance), I can make you a cloud instance for that. Or do you want to just skip that since you have an existing dev instance somewhere else? There still might be some use in testing copr builds and such...

Hi Kevin,

nowadays we have a testing instance based on Openshift here: developer-phracek.rhcloud.com .
I think Fedora would be better. There could be in future some cron jobs which will update a pages. Like version of languages, etc. But in the future.

I guess, that uploading a static content is the best solution instead of building a page directly on the "developer.fedoraproject.org" page.
developer-phracek.rhcloud.com is going to be used for testing in future like nowadays.

Would it be possible to add a reference from getfedora.org to this site?
Who is reposponsible for it?
Should be we create a new ticket?

Greetings
Petr

I created an update script that would just replace the part of the index page with new posts. The script depends on a few things that are already packaged in Fedora and could be run as a cron job or something.

I think we should aim for something like that now and deal with full rebuilds later (we can start the packaging work for Jekyll since it will take a long time). Would that be feasible from your point of view? Having Jekyll on the server is not necessary.

I talked with asamalik this morning, and we thought the best way forward would be:

• I'll setup a cloud instance (fedora 22).
• You can setup things on that instance.
• I'll point developer.fedoraproject.org to it when you are ready.

Note that this will be a single cloud instance, so it will be possibly down when we are doing maint on the cloud or if it runs into some serious problem.

Further ahead:

• You folks will work to package jekyll and it's needed deps.
• Once those are in place we can look at making a backend instance inside normal Fedora infrastructure that generates the site.
• That backend instance will push the static site to our network of proxies which will serve them to users.

This would mean there would almost never be any downtime (the content is on 11 proxies all over the world). Also content would be 'closer' to users so would be faster in many cases.

ok, developer.fedorainfracloud.org is live.

asamalik jstribny phracek should have their ssh keys in to login as root.

The instance is in Fedora Infrastructure ansible. if you like we can try and keep that up to date, or just leave it as doing the base setup and you manually setup on top of that as you prefer until you get things setup.

Let me know if we can assist with anything further and when you are ready to direct the developer.fedoraproject.org name to the instance.

We can keep this open, or just close it and file a new ticket for the domain and the later move to staging/production down the road?

Thanks!

I believe our goal is to have a 'git push' type of deploy in the future. Part of the deploy is also a simple Ruby script that fetches the Fedora Planet content[0]. This script should however not be run just when deploying, but also as a cron job regularly updating the index.html. If you can anyhow help us to achieve that, we would welcome your help.

[0] https://github.com/developer-portal/website/blob/master/rss.rb

We were thinking about the production deployment today and we came up with the following solution:

• A static website would be available in a Git Hub repository.

• We would install two scripts to the backend - as we talked about before:
  1) The first one would fetch the static content from the Git Hub repo, lets say, every day. This would be an easy shell script.

  2) The second one would refresh blog posts on the homepage every 15 minutes or so. This would be a ruby script.
  - Everything necessary to run these scripts is available in Fedora/Epel.
  - The backend would serve proxies - as we talked about before.

Would this be acceptable?

Is it possible to have this up and running on Friday, Oct 31?

Also, I assume that we'd write an Ansible playbook for deployment - the same way as for Copr (or other fedora infra stuff) deployment?

Would it be possible to write the script to update in python? The backend machines don't have any ruby on them currently and we all are much more python folks.

You could consult with the fedora websites team about that, they may already even have a solution since they update start.fedoraproject.org with magazine posts.

We are heading into freeze next tuesday, so anything after that will need a freeze break, but it should be possible to get it setup by the 31st I would think.

Yes, we will use ansible to set things up and deploy. We could deploy this to the existing 'sundries' box that already builds the other static websites and syncs to proxies (which will mean not too much setup hopefully as much of it would already be in place)

Hi all, what about getting this portal into the web repository? Ok, I saw you build the pages with jekyll and use ruby...but for static websites?

In our web repo we have static websites, but we use also python scripts to pull content from other sources (like fedoramagazine posts for start.fpo), but we have others too. In my understanding this solution would be much easier for all of us, for you (if you use our makefiles to build the pages you don't need jekyll), for Infra (a Cloud instance just for a static website?) and it's easier to do development, as we have already staging instances running.

I wrote the python script (it's not perfect but it works just fine) to pull the posts from fedmagazine, so you can just adapt it if you want [1], we just need to make the pages run with our makefiles. All websites are built hourly.

Last but not least, in the web-repo you can use genshi strings for I18n, I can setup the zanata source and add the developer portal in the websites project, in order to have it translated too. Thoughts?

[1] https://git.fedorahosted.org/cgit/fedora-web.git/tree/start.fedoraproject.org/build/magfeed.py

Hi Kevin and rubyduck,

Would it be possible to write the script to update in python? The backend machines don't have any ruby on them currently and we all are much more python folks.

It's already written and its in Ruby as the rest of the site is generated with Ruby so it does not make sense to have it in two different language stacks. Ruby is part of core Fedora so I don't see a problem of using it.

Yes, we will use ansible to set things up and deploy. We could deploy this to the existing 'sundries' box that already builds the other static websites and syncs to proxies

Here is a simple shell deploy script that automates it for us:
https://github.com/developer-portal/website/blob/master/deploy.sh

You can see what's necessary and update the scripts you use.

Ok, I saw you build the pages with jekyll and use ruby...but for static websites?

Yes, it helps to build the site as we want and Jekyll is probably the most popular static-site generator at the moment. The final site is just HTML files of course.

(if you use our makefiles to build the pages you don't need jekyll),

There is nothing else expect updates of the RSS feed.

I wrote the python script (it's not perfect but it works just fine) to pull the posts from fedmagazine, so you can just adapt it if you want

I already wrote it in Ruby, but if you decide to rewrite it in Python, feel free to do so. I don't see a reason to change a language stack if everything is in Fedora.

Last but not least, in the web-repo you can use genshi strings for I18n

I think we currently don't plan on having the site translated. It would be a lot and lot of work. Maybe one day when we see that the portal got really popular and useful.

As I don't know current processes and reasoning behind the only-Python environment or using just Makefiles to generate static sites I am sorry that I am not following.

First of all release Fedora23 is 27th Oct and not 31th Oct. And I guess that portal should be available in this date with reference from getfedora.org.
Releasing Fedora Developer Portal is really important from developer point of view.

Question is why to migrate it to Python?

But we would like to release a static pages and afterwards MAYBE we will migrate ruby script to python if don't mind. I think that it doesn't block to release first version of Fedora Developer Portal.
Replying to [comment:16 jstribny]:

Hi Kevin and rubyduck,

Would it be possible to write the script to update in python? The backend machines don't have any ruby on them currently and we all are much more python folks.

It's already written and its in Ruby as the rest of the site is generated with Ruby so it does not make sense to have it in two different language stacks. Ruby is part of core Fedora so I don't see a problem of using it.

Yes, we will use ansible to set things up and deploy. We could deploy this to the existing 'sundries' box that already builds the other static websites and syncs to proxies

Here is a simple shell deploy script that automates it for us:
https://github.com/developer-portal/website/blob/master/deploy.sh

You can see what's necessary and update the scripts you use.

Ok, I saw you build the pages with jekyll and use ruby...but for static websites?

Yes, it helps to build the site as we want and Jekyll is probably the most popular static-site generator at the moment. The final site is just HTML files of course.

(if you use our makefiles to build the pages you don't need jekyll),

There is nothing else expect updates of the RSS feed.

I wrote the python script (it's not perfect but it works just fine) to pull the posts from fedmagazine, so you can just adapt it if you want

I already wrote it in Ruby, but if you decide to rewrite it in Python, feel free to do so. I don't see a reason to change a language stack if everything is in Fedora.

Last but not least, in the web-repo you can use genshi strings for I18n

I think we currently don't plan on having the site translated. It would be a lot and lot of work. Maybe one day when we see that the portal got really popular and useful.

As I don't know current processes and reasoning behind the only-Python environment or using just Makefiles to generate static sites I am sorry that I am not following.

Here is a simple shell deploy script that automates it for us: ​https://github.com/developer-portal/website/blob/master/deploy.sh
You can see what's necessary and update the scripts you use.

ok, issues:

• Our sundries server is rhel7. There's no rubygem-liquid or rubygem-actionview available.

• Is there some way we could redo just the cron/rss updater to not use those for now? Or is it tied into the rest of the site?

• Alternately, can we get those things in EPEL?

• Or alternately, could we not have the blog updater for the initial release and add it later?

Replying to [comment:16 jstribny]:

I wrote the python script (it's not perfect but it works just fine) to pull the posts from fedmagazine, so you can just adapt it if you want

I already wrote it in Ruby, but if you decide to rewrite it in Python, feel free to do so. I don't see a reason to change a language stack if everything is in Fedora.

The question is not if it is or not in Fedora, but what we are using on the Fedora Infra-servers (EPEL). In this specific case it's not even packaged, that's why I was simply trying to understand why this came up in a language (Ruby and Jekyll) we are not using in Infra.

So the web Repo is not an option at this point, need to think about a dedicated solution.

Replying to [comment:18 nirik] and [comment:19 robyduck].

What about this:

1. We will rewrite the RSS refresh script to Python. Robyduck, if you have something already, I'll definitely have a look.
2. We will then deploy the generated static html website + the script in Python mentioned above. Nirik, would that be OK?
3. We will use this model for now, while:
4. We will work with the websites team to make the build process as good as possible. I used the word 'good' intentionally as I don't know what we'll come up with. I'm open to any discussion.

Our sundries server is rhel7. There's no rubygem-liquid or rubygem-actionview available.
The question is not if it is or not in Fedora, but what we are using on the Fedora Infra-servers (EPEL).

Then I am sorry I though you run Fedora. Now it makes sense... I wish we would realize that sooner, but the testing/staging server we got was Fedora.

So well, we could rewrite the RSS script to get rid of the dependencies, just using plain Ruby/EPEL, that would be okay with you?

Replying to [comment:20 asamalik]:

Sure, that sounds good to me.

Replying to [comment:21 jstribny]:

Sorry for the miscommunication. ;(

Well, the thing is not that we dislike ruby or jekyl or any particular packages. The thing is that once we deploy something we have to maintain it moving forward. Right now if something went wrong we could talk to you and get it fixed. In a few years you might move on to some other project and we then might have no one who knows how to fix things.

So, we prefer to stick with things where we have a large pool of developers that can assist and right now thats python and related tools. Does that make sense?

Replying to [comment:20 asamalik]:

Replying to [comment:18 nirik] and [comment:19 robyduck].

What about this:

1. We will rewrite the RSS refresh script to Python. Robyduck, if you have something already, I'll definitely have a look.
2. We will then deploy the generated static html website + the script in Python mentioned above. Nirik, would that be OK?
3. We will use this model for now, while:
4. We will work with the websites team to make the build process as good as possible. I used the word 'good' intentionally as I don't know what we'll come up with. I'm open to any discussion.

Cool! If you need any help or just some short advices on how the build process is done, I'll be happy to help.

Hi folks,

finally I have migrated rss.rb -> rss.py.
https://github.com/developer-portal/website/blob/master/rss.py

Greetings
Petr

Replying to [comment:23 robyduck]:

Replying to [comment:20 asamalik]:

Replying to [comment:18 nirik] and [comment:19 robyduck].

What about this:

1. We will rewrite the RSS refresh script to Python. Robyduck, if you have something already, I'll definitely have a look.
2. We will then deploy the generated static html website + the script in Python mentioned above. Nirik, would that be OK?
3. We will use this model for now, while:
4. We will work with the websites team to make the build process as good as possible. I used the word 'good' intentionally as I don't know what we'll come up with. I'm open to any discussion.

Cool! If you need any help or just some short advices on how the build process is done, I'll be happy to help.

So we will have the deployment with rss.py soon. Currently we use the following script:
https://github.com/developer-portal/website/blob/master/deploy.sh

If we get the production server, can we use the same script? Or will the prod. server setup differently? In that case we need exact steps/configuration we need to do/provide.

Well, we are ready for the release. We have the website + python script prepared. What's next?

And how the deployment is going to work? Would the backend fetch some git repo with the static website and the python script? Or are we going to get a push access somewhere? We have agreed that the push access would be a preferred option, but we are fine with both of the above.

Great. I think the next steps I can work on this week with you:

1. I don't think push is going to work, we don't have a good way to push to internal machines (most don't have any outside IP address mapping anyhow), so we will need to use pull I think.

So, one way this could work is we install rsyncd on your cloud instance and allow rsync:// of the site for our ip's. Then we can pull from there via rsync. Is that workable?

1. We should get things working in our staging instance.
   I can work on this, would just need to setup a cron to rsync pull the site and setup the cron to run the python rss job. This would be on sundries01.stg.

Then, we setup a cron on proxy01.stg that pulls from sundries01.stg and serves the live content. This would be 'https://developer.stg.fedoraproject.org' for the site.

1. Once all thats working we do the same things as step 1 in production and the site is live at https://developer.fedoraproject.org

Does that sound all good? or am I missing anything?

All right Kevin.

So you probably want a separate git repository with just the generated site and the rss.py script from us? Or?

We could do a git repo (then we could use git pull) or just the bare site (and then we could just use rsync).

Either way.

Great!

I have set up the repo with static content: https://github.com/developer-portal/static-generated.git
The rss python script is here: https://github.com/developer-portal/website/blob/master/rss.py

Edit: rewriting the comment to be more clear.

Kevin,

the final site is at:

https://github.com/developer-portal/developer.fedoraproject.org

(Please note I renamed it from static-generated).

RSS script is included in the repository at:
https://github.com/developer-portal/developer.fedoraproject.org/blob/master/rss.py

Please let us know if you need something else. Thank you.

Replying to [comment:29 kevin]:

We could do a git repo (then we could use git pull) or just the bare site (and then we could just use rsync).

Either way.

Good to hear that we are soon before an official release.
Do you have an ETA when the portal will be out?

Do we have an access to official Fedora Developer Portal? Like over ssh?

Or better say, from our point of view, we need to have only our https://github.com/developer-portal/developer.fedoraproject.org git repo.

Rest of work will be done by your side, right?

Greetings
Petr

ok. Can you all please check:

https://developer.stg.fedoraproject.org/

and see if it looks right, updates as you expect when you update the git repo (it only syncs once per hour, so you will have to be patient with changes).

If all that checks out I can submit a freeze break to bring it into production. Did you want to wait for any particular time for that? Or should we do it as soon as we are ready and you can announce things after that?

Replying to [comment:33 kevin]:

ok. Can you all please check:

https://developer.stg.fedoraproject.org/
Yeah, page seems to be right.

and see if it looks right, updates as you expect when you update the git repo (it only syncs once per hour, so you will have to be patient with changes).

If all that checks out I can submit a freeze break to bring it into production. Did you want to wait for any particular time for that? Or should we do it as soon as we are ready and you can announce things after that?

I guess, that we can release it during Wednesday or Thursday. But definitely till end of this week.

Would it be possible to get a logs from rss.py script? E.g. If some error occurs.
Greetings

| I guess, that we can release it during Wednesday or Thursday. But definitely till end of this week.

Well, like I said, we can enable it anytime and that doesn't have to be when we announce it. ;)

| Would it be possible to get a logs from rss.py script? E.g. If some error occurs. Greetings

Sure. it's a cron job, we can send it to whoever. Who would you like to get it?

Kevin,

I am all for releasing before announcement, but hold on as we should fix the rss.py script first to strip all HTML tags.

So we tested it and fixed the issues with rss.py. Now it should be alright and I think we are ready to move it to the production.

One last question though, I expected the rss.py script to be moved outside the exposed site (to not be accessible from https://developer.stg.fedoraproject.org/rss.py). It's not a big issue but I think it would be better.

Sure, we can move it if you like.

If you prefer we could just put it in our ansible repo and install it on the machine. However, then you will need to go via us to make changes to it. That might make the most sense from a security standpoint anyhow.

I'll look at making that change and submit a freeze break to move things live in production soon. ;)

I've done those things.

Waiting on +1s to my freeze break now. ;)

This went live earlier today and should be up and running in production at https://developer.fedoraproject.org/ ;)

We should try and have a meeting with you folks, the websites team and infrastructure after the release and discuss plans moving forward to streamline things. ;)

Thanks.

Kevin, the staging takes our changes, but the production has wrong version of rss.py and the site is broken (as you can see).

Replying to [comment:40 kevin]:

This went live earlier today and should be up and running in production at https://developer.fedoraproject.org/ ;)

We should try and have a meeting with you folks, the websites team and infrastructure after the release and discuss plans moving forward to streamline things. ;)

Thanks.

Thanks for releasing.

And as I have commented a while ago. If we have access to developer.fedoraproject.org machine, then we can correct it really soon.

What do you think about it kevin?

Greetings

Petr

Edit: No it is broken, sorry.

We should try and have a meeting with you folks, the websites team and infrastructure after the release and discuss plans moving forward to streamline things. ;)

And yes, I agree, we should meet.

Replying to [comment:42 phracek]:

Thanks for releasing.

And as I have commented a while ago. If we have access to developer.fedoraproject.org machine, then we can correct it really soon.

What do you think about it kevin?

There isn't such a machine. ;)

There's a backend (sundries01.phx2.fedoraproject.org) and 13 proxies spread all around the world with the content (proxy01,... proxy13)

The issue you were seeing turned out not to be the rss.py, but the way it was syncing out to the proxies. ;(

I hopefully have it all fixed now... do let me know if you see any issues.