Jürgen Gmach

Am I famous?

written by Jürgen Gmach on 2019-05-29

A couple of months ago, Brett Cannon, one of the better known Python core developers, announced on Twitter that he contributed to over 100 open source projects.

first reaction

Woohah! Congratulations!

second reaction

How did he know?

Everytime I get a PR merged I record the repo's URL for a tool I created to analyze my contributions to OSS.

Ah... k. I did not do that.

third reaction

To how many open source projects did I contribute? Well, I only started contributing recently, like one year ago or so... and as I said, I did not take any notes... but luckily nowadays like most open source development happens on Github. As far as I remember I only contributed to a couple projects which are not present on Github, and that was for batou and Roundcube.

So, if only Github provided some API to query the data it stored about me... well, it does.

say hello to graphql

GraphQL is a query language for web APIs. It was developed by Facebook, and according to rumors it will supersede REST in the near future.

Github kindly not only offers an API, but also an in-browser IDE (based on GraphiQL) to explore the data.

say hello to graphiql

When you first hit Github's instance of GraphiQL ( https://developer.github.com/v4/explorer/ ), you will see something like this:

Alt-Text

So, obviously, first you have to login with your Github credentials (green "Sign in with Github" button in the top right).

first encounter

When I first saw this page, I played around a bit and left - as I did not get what I wanted. Luckily, at last weekend's PyConWeb19, Arthur Bayr did a workshop on GraphQL and Graphene - thanks a ton!

Now I am ready to explore my data!

overview

On the top left, there is room to enter the query.

On the right, the result will be shown.

On the far right, there is the small, but super helpful Docs button.

On the bottom left is something I did not use yet.

iteration one

After you have logged in, leave the query as is...

query { 
  viewer { 
    login
  }
}

... and just hit the play button above the "query field".

You should get a result similar to mine - but with your username

{
  "data": {
    "viewer": {
      "login": "jugmac00"
    }
  }
}

explanation

How do I know this?

On the far right, there is the Docs button. Small button, but lots and lots of information! Actually, you can explore the complete API when you open the docs and follow the links.

iteration two

Either by reading the docs or by hitting CTRL+SPACE which triggers autocompletion when you are in the query box, you will eventually come across a subfield with name repositoriesContributedTo - that sounds fine!

NOTE:
You cannot query arbitrary data - all fields, subfields, parameters... have to be defined by the API provider.

When you just enter the following...

query { 
  viewer {
    repositoriesContributedTo
  } 
}

... right below repostoriesContributetTo there will be a red squiggly line. When hover over the word, you can read "Field .. must have a selection of subfields".

Whatever! The cool thing.. if you hit the play button anyway, the IDE autocompletes your query to...

query { 
  viewer {
    repositoriesContributedTo {
      edges {
        node {
          id
        }
      }
    }
  } 
}

The result is... hm... an error message: "You must provide a first or last value to properly paginate the repositoriesContributedTo connection."

Ok, let's do this.

iteration three

Generally speaking, it is a very good idea to force pagination on all APIs, which return lists - another thing I learned at PyConWeb19 - no slides, but there is a video from another conference: Tech bankruptcy: Looking back on a decade of bad decision making

The mentioned value is actually a parameter, and parameters are put into parenthesis behind a (sub)fieldname, just like...

query { 
  viewer {
    repositoriesContributedTo(first: 100) {
      edges {
        node {
          id
        }
      }
    }
  } 
}

If you only type first, you get a friendly reminder to provide a number, which must not be greater than 100.

The result...

{
  "data": {
    "viewer": {
      "repositoriesContributedTo": {
        "edges": [
          {
            "node": {
              "id": "MDEwOlJlcG9zaXRvcnk2OTI2NTQ="
            }
          },
...

... does not look to bad, but not very human readable.

iteration four

If you replace id with nameWithOwner (via API exploration STRG+SPACE or from the docs), you'll get much better data:

query { 
  viewer {
    repositoriesContributedTo(first: 100) {
      edges {
        node {
          nameWithOwner
        }
      }
    }
  } 
}

The result...

{
  "data": {
    "viewer": {
      "repositoriesContributedTo": {
        "edges": [
          {
            "node": {
              "nameWithOwner": "python-babel/flask-babel"
            }
          },
          {
            "node": {
              "nameWithOwner": "malthe/chameleon"
            }
          },
...

But we want the total number of contributions.

iteration five

As I learned at the workshop, either the server or the client has to implement some count or sum functions, as you can only query what is offered (or process it client side).

So, let's have another look, what repositoriesContributedTo has to offer...

If you click on repositoriesContributedTo and then on RepositoryConnection! in the Docs, you'll see totalCount - nice! And also, you'll see nodes - which can simplify the "first get all edges and then the corresponding nodes" syntax.

query { 
  viewer {
    repositoriesContributedTo(first: 100) {
        totalCount
        nodes {
          nameWithOwner
      }
    }
  } 
}

The result...

{
  "data": {
    "viewer": {
      "repositoriesContributedTo": {
        "totalCount": 66,
        "nodes": [
          {
            "nameWithOwner": "python-babel/flask-babel"
          },
          {
            "nameWithOwner": "malthe/chameleon"
          },
...

Very nice.. except... also proprietary (private) repositories are listed...

iteration six

Visiting the docs again, repositoriesContributedTo offers a privacy parameter.

query { 
  viewer {
    repositoriesContributedTo(first: 100, privacy: PUBLIC) {
        totalCount
        nodes {
          nameWithOwner
      }
    }
  } 
}

The final result...

{
  "data": {
    "viewer": {
      "repositoriesContributedTo": {
        "totalCount": 54,
        "nodes": [
          {
            "nameWithOwner": "python-babel/flask-babel"
          },
          {
            "nameWithOwner": "malthe/chameleon"
          },
...

what's left

First, once again, thank you very much Arthur for teaching me the basics of GraphQL!

Now it is up to the reader to explore the API further... oh wait.. what is the includeUserRepositories parameter doing...

Damn! I missed to include my very own projects! :-)

Came for the talks, stayed for the raffles

written by Jürgen Gmach on 2019-05-27

summary of pyconweb 2019

F... awesome!

talks and workshops

I tried to collect all slides from Twitter and Telegram - in case I missed some, please drop me a line @jugmac00.

Saturday, 25 May 2019

Sunday, 26 May 2019

post scriptum

Thanks for the Echo Dot! I have not yet dared to open the pandora's box, but I will!

updates

2019-06-03

Chrome no longer shows saved passwords

written by Jürgen Gmach on 2019-05-20

Concerning security and privacy, you either can get stalled because you can't get it perfect right from the beginning, or you improve over time.

I prefer the latter one. And concerning password security I go with Troy Hunt: Password managers don't have to be perfect, they just have to be better than not having one.

And actually the password manager of a browser is already a password manager. I would not use it for very important passwords, but it is absolutely fine for those dozens of dozens forum logins and so on. Currently, I still manage the more important passwords with VeraCrypt - which is not that comfortable, but does its job.

works fine except when it does not

So, Google Chrome's password manager works fine for me... except when it does not.

Chrome stopped showing the saved passwords - from one day to another. I can't recall whether it was the Fedora 28 -> 29 Upgrade or maybe a Chrome update.

Anyway - it is extremely annoying to lookup passwords for every forum and and other sites which requires a login.

the easy fix

As I sync everything my profile with Google (hum, this sounds like something I should reevaluate some time) it is just a matter of deleting the broken profile and restarting Chrome.

$ cd /home/jugmac00/.config
$ rm -rf google-chrome

It is as simple as that. And sure, I experience many more rough edges since I migrated to a Linux desktop, but I always feel like I keep the control and every problem is very easily fixable.

Recap day three and wrap up of "Zope spring cleaning"

written by Jürgen Gmach on 2019-05-10

This blog post is work in progess... much more to come - but the train is only 10 minutes late (uncommon in Germany :-) ) , so I am not able finish the blog post now.

tl/dr - what a ride!

progress on day 3

The main effort of everybody at the sprint was to wrap up all loose ends, finalize what one started, and of course get Zope 4 final released!

As there was no final standup, I try to list what I picked up...

Again, Jens Vagelpohl, Marius and Dieter and maybe some more guys I possibly missed, supported the sprint massively from remote!

more news

There are some more news about merging the Zope into the Plone foundation.

Recap day two of "Zope spring cleaning"

written by Jürgen Gmach on 2019-05-10

Very brief recap, as I want to spend the remaining time with some more documentation updates.

progress

Again, Jens Vagelpohl, Dieter, Marius and Jason very actively contributed from remote.

P.S.: I will update this blog post when I sit in the train back home... let's get Zope 4 ready to be released!

updates

2019-05-10

Recap day one of "Zope spring cleaning"

written by Jürgen Gmach on 2019-05-09

More infos about the sprint

progress

Also, Jens Vagelpohl and Dieter Maurer contributed a lot by adding valuable insights to some complicated discussions, reviewed a lot of pull requests... Jens also added documentation about how to setup daemoize Zope 4 on a systemd powered Linux box and fixed on old bug with importing ZEXP files.

Some more positive news...

Matthew Wilkes announced some updates on the upcoming merger of the Zope into the Plone foundation, and thus the improvements on the onboarding for new developers concerning the committer agreement.

other than that...

We have plenty of fun here. The venue ( https://twitter.com/koffij_halle ) is great - the food is excellent! Basti, the new cook, really rocks the kitchen!

Face to face with "Earl Zope" and his henchmen

written by Jürgen Gmach on 2019-05-03

NOTE: This blog post is WORK IN PROGRESS and will be updated regularly. Though, this post is first and foremost written for myself, please do not hesitate to drop me a line at juergen dot gmach at googlemail com or directly contribute to this article via GitHub.com.

This blog post tries to ease beginning and keeping up with development of the open source web application server Zope and its universe.

what makes it hard to start contributing to Zope

prerequisites

Zope and probably the most of its related packages are licensed under the ZPL. In order to contribute to the project, you need to sign a contributor's agreement. If you do not get a reply to your application in a reasonable amount of time, please resend your email.

development process

Once your application has been accepted, which is a mere formality, you may to start contributing.

where to find the source code

Zope and all its related packages, which reside under the umbrella of the Zope Foundation, are hosted at GitHub.

For instance, the repository for Zope is located at https://github.com/zopefoundation/Zope

NOTE: There are plans to merge the Zope Foundation into the Plone Foundation - current status is unknown.

flow

Though not 100% consistent, usually you cannot and you should not directly work on the master branch.

The recommended way is to clone the repository and work on a new branch.

Once finished, push your branch to the repository and create a pull request. Both your branch and the pull request will get automatically tested by TravisCI (see below).

Most repositories need both green tests and a successful review before a pull request gets merged.

Usually you will merge the branch yourself - in order to show off that you have signed the contributor agreement.

After the pull request had been merged, please delete your branch.

NOTE: If you want to implement very small changes - e.g. fixing typos - you can directly edit the file in your browser by clicking the pencil icon when viewing the file on GitHub.com.

tests

It is highly recommended that all code changes, bug fixes and new features should be tested by automatic tests.

Unlike most other Python projects, Zope/universe does not make use of pytest - which can be both quite surprising and a bit of a hurdle.

Zope/universe makes heavily use of zope.testrunner.

supported python versions

Currently, Python 2.7, 3.5, 3.6, 3.7 and 3.8 (dev) are supported for most of the Zope 4/universe packages. This reflects the officially supported Python versions.

Some packages also support PyPy.

Afaik, there are plans to drop support for Python 2.7 with Zope 5.

When adding support for a new Python version, or dropping support for an old one, you have to update several files:

continuous integration - mostly via travisci

Whenever you push to a branch or make a pull request, TravisCI automatically pulls your changes and runs the project's test suite.

This is configured via a .travis.yml file at root level of the repository.

NOTE: For packages, which should be supported on the Windows plattform, AppVeyor gets used.

travisci configuration

The configuration for TravisCI changes on quite a regular basis. There are OS upgrades, new and deprecated Python versions, syntax changes and also new and deprecated features for Travis itself.

Depending on how the tests are run, whether there are checks for styles or coverage, the .travis.yml can be quite different for different packages.

You can have a look at the recently updated configuration for zExceptions as a good example with up-to-date best practices.

deprecated travisci configuration

If you happen to see a sudo: false/true option in a .travis.yml file, you can safely remove it (cf. https://blog.travis-ci.com/2018-11-19-required-linux-infrastructure-migration).

travisci trouble shooting

1) Information by Dieter Maurer:

The Python 3.8 problem was (likely) discussed in the Python mailing list today: the newest Python version has changed sys.implementation.cache_tag which prevents Python from finding shared objects installed for earlier versions. "pip" on the other hand seems not to take thin into account and thinks that the older version is still usable. Clearing the cache should fix the problem.

updates

2019.05.03

2019.05.04

2019.05.06

2019.05.08

it's never too late to join the party!

written by Jürgen Gmach on 2019-05-02

Although I started writing code already at the age of 13 (Generation C64), and I am a professional web and backend developer since 2007, I only started contributing to open source software quite recently - apart from the occasional bug report.

what is open source software?

Quote from Wikipedia:

Open-source software (OSS) is a type of computer software in which source code is released under a license in which the copyright holder grants users the rights to study, change, and distribute the software to anyone and for any purpose.

interesting tidbit

The term was coined by Tim O'Reilly, Eric Raymon and Bruce Perens in 1998 - that's just 20 years!

why care about open source software?

The company, I currently work for, makes heave use of open source sofware - maybe not right from the beginning, as the company predates the term open source software, but at least from the very start of the internet-age.

utilized open source software (excerpt)

why would you not contribute to open source software?

turning point

... but ... Zope is open source software!

Despite its age maturity Zope is still widely used, indirectly by the very popular and rock stable CMS Plone, but also for big custom applications, for instance union.cms, which gets used by ver.di and DGB.

That means that there is a broad interest of companies and unions alike to keep Zope going.

Focussing on the core competence, it is not always the best idea to employ software engineers yourself - at least not to build on the basis technology.

Luckily, there are quite some consultants out there working with the Zope / Plone universe, e.g. gocept gmbh & co. kg, which get hired by companies and the mentioned unions, in order to work on Zope. gocept on the other hand also has great interest in Zope's future, and therefore also heavily contributes back to the Zope universe, both with code contributions, but they also host so called sprints.

Sounds like a win win win situation!

sprint what?

A sprint is a set period of time during all kind of software developers come together to work on a certain project. In this case, the migration of Zope from Python 2.7 to Python 3.

history of Zope sprints @gocept

open source is fun and you'll learn a lot!

... but also historical facts ... src/ZPublisher/HTTPResponse.py

# Note that as of May 98, IE4 ignores cookies with
# quoted cookie attr values, so only the value part
# of name=value pairs may be quoted.

one last thing

Contributing to open source software should not exclusively be at the expense of your free time!

Ask your boss - today! Eventually, you'll be surprised - so was I.

notes

I presented this topic as a lightning talk at Mobile Stammtisch/GDG Regensburg on 20. December 2018.

ignore requests for apple touch icons

written by Jürgen Gmach on 2018-07-28

the problem

My applications error log is polluted with tracebacks of notFoundErrors from Apple browsers requesting apple-touch-icon.pngs.

For those, who do not know... Touch icons are to Apple's mobile devices what favicons are to (desktop) web browsers.

According to ComputerHope:

When someone bookmarks your web page or adds your web page to their home screen, this icon is used.

So, when you run a public website, no question, you want to put some effort into making some great icons, so your website looks great on your users' home screen and bookmark section.

And by some effort I mean quite some. I was not able to quickly find out how many icons one has to create currently - for all different screens sizes and resolutions.

If you have to take care of touch icons, try Apple's Configuring Web Applications as a starting point.

initial situation

I do not run a public website, but an internal application with a two digit user number.

When one of my colleagues accesses the application with an Apple device, and does "things", nginx, configured as reverse proxy, passes the request for the touch icon to the application server, and it makes boom!

Well, the home screen will be full - at some time, but I guess quite some bookmarks are made, and thus my application's error log gets spammed with messages I do not care about - at least for the moment, but of course, I do not want to see them.

technical solution >> design solution

The easiest way to get rid of those messages, is NOT to create estimated 18 (update: 40) differently named and sized touch icons, but not to pass those requests to the application server.

nginx makes this easy, as all I needed to add to the configuration was:

location ~ /apple-touch-icon(|-\d+x\d+)(|-precomposed).png {
    return 404;
    log_not_found off;
}

where...

apple-touch-icon.png
apple-touch-icon-precomposed.png
apple-touch-icon-74x74.png
apple-touch-icon-180x180.png

Fair enough?

bonus

Want to be prepared for the glorious time when you introduce touch icons?

Dig into nginx' try_files directive , so nginx looks for a file and delivers it, and if it's not there, you can return a 404 (and not pass the request to your application server).

Following (untested) configuration snippet should do the trick:

location ~ /apple-touch-icon(|-\d+x\d+)(|-precomposed).png {
    try_files $uri =404;
}

further resources

sudo dnf install beautiful-problems

written by Jürgen Gmach on 2018-07-04

There is something I immediately disliked when using Fedora the first time - the Bash prompt.

For many years I've been using a fixed prompt at work for all my machines, a beauty in green and yellow...

And when I first started Bash after installing Fedora on my home laptop, I encountered this grey monotony...

[jugmac00@localhost ~]$

I did not try to recreate the prompt I have at work, but I wanted to try something new - and oh my zsh, ah, oh my god, zsh did not come to my mind - yuck!

the good (looking prompt and statusline)

Instead, while googling for resources about Fedora, I stumbled upon Add power to your terminal with powerline published on https://fedoramagazine.org/.

From Powerline's GitHub page:

Powerline is a statusline plugin for vim, and provides statuslines and prompts for several other applications, including zsh, bash, fish, tmux, IPython, Awesome, i3 and Qtile.

The article is written very well, and easy to follow. A few minutes later, installing a few packages here, adding some lines to .bashrc and .vim.rc there, both Bash and Vim looked great!

bash

Alt-Text

vim

Alt-Text

the bad (experience)

The wow-effect lasted only a couple of minutes, until I tried to work on a Python project.

When you work on a Python project, it's an hell of an idea to use virtualenv instead of installing dependencies system wide.

From virtualenv's documentation page:

virtualenv is a tool to create isolated Python environments.

So in order to not mess up with Python based Linux applications - such as Powerline - I activated the virtualenv for my project, started Vim... and it made booooom!

Error detected while processing /home/jugmac00/.vimrc:
line   36:
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'powerline'
line   37:
Traceback (most recent call last):
  File "<string>", line 1, in <module>
NameError: name 'powerline_setup' is not defined
line   38:
Traceback (most recent call last):
  File "<string>", line 1, in <module>
NameError: name 'powerline_setup' is not defined
Traceback (most recent call last):
  File "<string>", line 4, in <module>
ModuleNotFoundError: No module named 'powerline'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "<string>", line 9, in <module>
ModuleNotFoundError: No module named 'powerline'
An error occurred while importing powerline module.
This could be caused by invalid sys.path setting,
or by an incompatible Python version (powerline requires
Python 2.6, 2.7 or 3.2 and later to work). Please consult
the troubleshooting section in the documentation for
possible solutions.
Unable to import powerline, is it installed?
Press ENTER or type command to continue

As a Python developer I am used to see tracebacks, so I was not scared away immediately, and this traceback was even a very informative one:

ModuleNotFoundError: No module named 'powerline'

In order to verify the missing package, I left Vim and started a Python interpreter and tried to import the Powerline package manually...

Here we go! Having a look at the longish traceback from above again, I also spotted the exact location of the problem, the import statement in .vimrc.

python3 from powerline.vim import setup as powerline_setup
python3 powerline_setup()
python3 del powerline_setup

There are only two Python 3 interpreters on my system (that I know of)

So obviously, Vim is using my project's environment - instead of the system one!

the ugly (hack)

Two possible solutions came to my mind:

As the few lines of code from above suggest, it seems to be possible to write ordinary Python code within .vimrc.

But where is Powerline's Python package?

Say hello to Python's ability for introspection...

jugmac00 ~> python3
Python 3.6.5 (default, Mar 29 2018, 18:20:46) 
[GCC 8.0.1 20180317 (Red Hat 8.0.1-0.19)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import powerline
>>> powerline.__path__
['/usr/lib/python3.6/site-packages/powerline']

So I just had to add the following line to .vimrc:

python3 import sys;sys.path.append("/usr/lib/python3.6/site-packages/")

And here we go, Powerline and Vim, united once again - and this time, even when using an activated virtualenv.

drawback

Adding system-wide site-packages could have some unforseen side effects, but at least it only effects Vim, and not my project. And, well... yolo :-)

bug report

Googling for the error message did not reveal much more information about this problem. But at least I found a bug report at Powerline's issue tracker - yet without an answer.

Lately, Powerline's development seems to be not that active anyway, and to be honest, I am not sure, whether this is a Powerline or maybe even a Vim bug.