Github List All Repositories in Organization or Personal using REST API

I have been trying to get the complete list of Repositories private and public in my github account ( user + Organization ) using GITHUB rest api

I had to create a CSV report of  All repositories in a specific format like shown below.

I have done similar thing earlier with bitbucket and the script is available here for your reference

Before we go to the script some background on the script and the prerequisites.

 

Prerequisites

• JQ - Json parser
• Linux ( Mac is fine ) with following commands available
  • awk
  • sed
  • bash
  • curl
  • tr
• permission to create and run a script and a temp file

 

How to use the script

Copy the script code and paste it into your editor like vi/nano/code etc and save it with .sh extension.

My preferred name for this script is GitRepoList.sh

Set it to executable permission with some command like

chmod a+x GitRepoList.sh

Update the script with your corresponding details such as org name, token, username etc

Then Run the script and redirect the output to a CSV file name of your choice

./GitRepoList.sh > Output.csv

 

Script to List All repositories in Github using REST API

Here is the shell script that uses all the aforementioned linux commands along with curl and jq parser to create a neat CSV report of All repositories.

#!/bin/bash
# Author: Sarav AK
# Email: [email protected]
# Created Date: 19 Aug 2021
# 

USERNAME=<Your_Github_Username>
TOKEN=<Github_API_token>

# No of reposoitories per page - Maximum Limit is 100
PERPAGE=100

# Change the BASEURL to  your Org or User based
# Org base URL
BASEURL="https://api.github.com/orgs/<your_org_name>/repos"

# User base URL
# BASEURL="https://api.github.com/user/<your_github_username>/repos"

# Calculating the Total Pages after enabling Pagination
TOTALPAGES=`curl -I -i -u $USERNAME:$TOKEN -H "Accept: application/vnd.github.v3+json" -s ${BASEURL}\?per_page\=${PERPAGE} | grep -i link: 2>/dev/null|sed 's/link: //g'|awk -F',' -v  ORS='\n' '{ for (i = 1; i <= NF; i++) print $i }'|grep -i last|awk '{print $1}' | tr -d '\<\>' | tr '\?\&' ' '|awk '{print $3}'| tr -d '=;page'`

i=1

until [ $i -gt $TOTALPAGES ]
do
  result=`curl -s -u $USERNAME:$TOKEN -H 'Accept: application/vnd.github.v3+json' ${BASEURL}?per_page=${PERPAGE}\&page=${i} 2>&1`
  echo $result > tempfile
  echo "Repo Name, SSH URL, Clone URL"
  cat tempfile|jq '.[]| [.name, .ssh_url, .clone_url]| @csv'|tr -d '\\"'
  ((i=$i+1))
done

 

This can be used for both private and public github repositories

This can be used for both user based accounts and Organization based accounts

All you have to do is to update the necassary details such as

• username
• git api token
• how many repositories to fetch per page ( maximum 100 allowed per page)
• base api url
  • For Organization Listing

    BASEURL="https://api.github.com/orgs/<your_org_name>/repos"

  • For Personal User account Listing

    BASEURL="https://api.github.com/user/<your_github_username>/repos"

 

 

How to create a Personal access or API token in Github

Follow this article from Github for more detailed info on how to create your personal access token

https://docs.github.com/en/github/authenticating-to-github/keeping-your-account-and-data-secure/creating-a-personal-access-token

Further customizations and updates

Feel free to fork this script  and add updates

Here is the list of information that github API provides for a Single repository. Hope this help in modifying this script

{
    "id": 171173763,
    "node_id": "MDEwOlJlcG9zaXRvcnkxNzExNzM3NjM=",
    "name": "SampleWebApp",
    "full_name": "AKSarav/SampleWebApp",
    "private": false,
    "owner": {
      "login": "AKSarav",
      "id": 17838978,
      "node_id": "MDQ6VXNlcjE3ODM4OTc4",
      "avatar_url": "https://avatars.githubusercontent.com/u/17838978?v=4",
      "gravatar_id": "",
      "url": "https://api.github.com/users/AKSarav",
      "html_url": "https://github.com/AKSarav",
      "followers_url": "https://api.github.com/users/AKSarav/followers",
      "following_url": "https://api.github.com/users/AKSarav/following{/other_user}",
      "gists_url": "https://api.github.com/users/AKSarav/gists{/gist_id}",
      "starred_url": "https://api.github.com/users/AKSarav/starred{/owner}{/repo}",
      "subscriptions_url": "https://api.github.com/users/AKSarav/subscriptions",
      "organizations_url": "https://api.github.com/users/AKSarav/orgs",
      "repos_url": "https://api.github.com/users/AKSarav/repos",
      "events_url": "https://api.github.com/users/AKSarav/events{/privacy}",
      "received_events_url": "https://api.github.com/users/AKSarav/received_events",
      "type": "User",
      "site_admin": false
    },
    "html_url": "https://github.com/AKSarav/SampleWebApp",
    "description": "A Sample Web Application with Snoop Servlet for Tomcat, Weblogic, Jboss, Websphere",
    "fork": false,
    "url": "https://api.github.com/repos/AKSarav/SampleWebApp",
    "forks_url": "https://api.github.com/repos/AKSarav/SampleWebApp/forks",
    "keys_url": "https://api.github.com/repos/AKSarav/SampleWebApp/keys{/key_id}",
    "collaborators_url": "https://api.github.com/repos/AKSarav/SampleWebApp/collaborators{/collaborator}",
    "teams_url": "https://api.github.com/repos/AKSarav/SampleWebApp/teams",
    "hooks_url": "https://api.github.com/repos/AKSarav/SampleWebApp/hooks",
    "issue_events_url": "https://api.github.com/repos/AKSarav/SampleWebApp/issues/events{/number}",
    "events_url": "https://api.github.com/repos/AKSarav/SampleWebApp/events",
    "assignees_url": "https://api.github.com/repos/AKSarav/SampleWebApp/assignees{/user}",
    "branches_url": "https://api.github.com/repos/AKSarav/SampleWebApp/branches{/branch}",
    "tags_url": "https://api.github.com/repos/AKSarav/SampleWebApp/tags",
    "blobs_url": "https://api.github.com/repos/AKSarav/SampleWebApp/git/blobs{/sha}",
    "git_tags_url": "https://api.github.com/repos/AKSarav/SampleWebApp/git/tags{/sha}",
    "git_refs_url": "https://api.github.com/repos/AKSarav/SampleWebApp/git/refs{/sha}",
    "trees_url": "https://api.github.com/repos/AKSarav/SampleWebApp/git/trees{/sha}",
    "statuses_url": "https://api.github.com/repos/AKSarav/SampleWebApp/statuses/{sha}",
    "languages_url": "https://api.github.com/repos/AKSarav/SampleWebApp/languages",
    "stargazers_url": "https://api.github.com/repos/AKSarav/SampleWebApp/stargazers",
    "contributors_url": "https://api.github.com/repos/AKSarav/SampleWebApp/contributors",
    "subscribers_url": "https://api.github.com/repos/AKSarav/SampleWebApp/subscribers",
    "subscription_url": "https://api.github.com/repos/AKSarav/SampleWebApp/subscription",
    "commits_url": "https://api.github.com/repos/AKSarav/SampleWebApp/commits{/sha}",
    "git_commits_url": "https://api.github.com/repos/AKSarav/SampleWebApp/git/commits{/sha}",
    "comments_url": "https://api.github.com/repos/AKSarav/SampleWebApp/comments{/number}",
    "issue_comment_url": "https://api.github.com/repos/AKSarav/SampleWebApp/issues/comments{/number}",
    "contents_url": "https://api.github.com/repos/AKSarav/SampleWebApp/contents/{+path}",
    "compare_url": "https://api.github.com/repos/AKSarav/SampleWebApp/compare/{base}...{head}",
    "merges_url": "https://api.github.com/repos/AKSarav/SampleWebApp/merges",
    "archive_url": "https://api.github.com/repos/AKSarav/SampleWebApp/{archive_format}{/ref}",
    "downloads_url": "https://api.github.com/repos/AKSarav/SampleWebApp/downloads",
    "issues_url": "https://api.github.com/repos/AKSarav/SampleWebApp/issues{/number}",
    "pulls_url": "https://api.github.com/repos/AKSarav/SampleWebApp/pulls{/number}",
    "milestones_url": "https://api.github.com/repos/AKSarav/SampleWebApp/milestones{/number}",
    "notifications_url": "https://api.github.com/repos/AKSarav/SampleWebApp/notifications{?since,all,participating}",
    "labels_url": "https://api.github.com/repos/AKSarav/SampleWebApp/labels{/name}",
    "releases_url": "https://api.github.com/repos/AKSarav/SampleWebApp/releases{/id}",
    "deployments_url": "https://api.github.com/repos/AKSarav/SampleWebApp/deployments",
    "created_at": "2019-02-17T21:19:55Z",
    "updated_at": "2020-11-02T08:11:30Z",
    "pushed_at": "2021-01-18T15:02:45Z",
    "git_url": "git://github.com/AKSarav/SampleWebApp.git",
    "ssh_url": "[email protected]:AKSarav/SampleWebApp.git",
    "clone_url": "https://github.com/AKSarav/SampleWebApp.git",
    "svn_url": "https://github.com/AKSarav/SampleWebApp",
    "homepage": null,
    "size": 18039,
    "stargazers_count": 1,
    "watchers_count": 1,
    "language": "Java",
    "has_issues": true,
    "has_projects": true,
    "has_downloads": true,
    "has_wiki": true,
    "has_pages": false,
    "forks_count": 42,
    "mirror_url": null,
    "archived": false,
    "disabled": false,
    "open_issues_count": 1,
    "license": null,
    "forks": 42,
    "open_issues": 1,
    "watchers": 1,
    "default_branch": "master"
  }

If you want to add any more fields to the csv like created_at, pushed_at  from the available fields given above.

You can update the array of jq like this, in the script

jq '.[]| [.name, .ssh_url, .clone_url, .created_at, .pushed_at]

Conclusion

Hope this helps and if you have any better ways to do this or written such script in any other language and wants to share with us. please post it in comments.

 

Cheers

Sarav AK

---