Jamie Tanna (https://www.jvt.me)
As I wrote in the post Analysing our dependency trees to determine where we should send Open Source contributions for Hacktoberfest:
In recent years, it has become unavoidable to build software on top of Open Source. This is absolutely a great thing, and allows developers to focus on fewer areas of domain specialisation as possible, as well as allowing a much wider range of users to pick up on defects and bring new features to our tools.
As I wrote in the post Analysing our dependency trees to determine where we should send Open Source contributions for Hacktoberfest:
However, with events such as the Log4Shell security vulnerability, times where maintainers have removed their libraries from package and source repositories, sometimes in political protest, it's understandable that businesses are somewhat hesitant about the sustainability of projects.
💖
Do you fully appreciate the depth of your dependency on the software supply chain?
Being able to understand how your business uses Open Source is really important for a few other key reasons:
How can we do it?
💰🤑💸
Let's use Open Source!
/usr/bin/whoami
Dependency Management Data (DMD) - dmd.tanna.dev
What's in the project?
dmd
dmd-web
dmd
dmd-web
SQLite database
commit 73a99614a2af6fa9f66508bab8541ed65e18ed66
Author: Jamie Tanna <>
Date: Thu Feb 2 09:23:43 2023 +0000
Initialise project
LICENSE.md | 13 +++++++++++++
README.md | 7 +++++++
public/index.html | 90 ++++++++++++++++++++++++++++++++++++
3 files changed, 110 insertions(+)
# produce some data that DMD can import, for instance via renovate-graph
npx @jamietanna/renovate-graph@latest --token $GITHUB_TOKEN your-org/repo
# set up the database
dmd db init --db dmd.db
# import renovate-graph data
dmd import renovate --db dmd.db 'out/*.json'
# optionally, generate advisories
dmd db generate advisories --db dmd.db
# then you can start querying it
sqlite3 dmd.db 'select count(*) from renovate'
Currently:
renovate-graph
dependabot-graph
endoflife-checker
Pre-baked, Open Source'd queries:
$ dmd report --help
advisories Report advisories that are available for packages or dependencies in use
golangCILint Query usages of golangci-lint, tracked as a source-based dependency
mostPopularDockerImages Query the most popular Docker namespaces and images in use
mostPopularPackageManagers Query the most popular package managers in use
Right now we can ask:
What if we could ask:
Dependency advisory data sources:
AWS infrastructure advisory data sources:
🤫
Custom advisories 🦸
Community provided advisories via -contrib:
INSERT INTO advisories (
package_pattern,
package_manager,
version,
version_match_strategy,
advisory_type,
description
) VALUES (
'github.com/golang/mock',
'gomod',
NULL,
NULL,
'UNMAINTAINED',
'golang/mock is no longer maintained, and active development been moved to github.com/uber/mock'
);
dmd owners
+ owners
table to the rescue!select
distinct
renovate.platform,
renovate.organisation,
renovate.repo,
owner
from
renovate
left join owners
on renovate.platform = owners.platform
and renovate.organisation = owners.organisation
and renovate.repo = owners.repo
I.e. aws-lambda-go
package (example)
select
distinct renovate.platform,
renovate.organisation,
renovate.repo,
version,
owner
from
renovate
left join owners on renovate.platform = owners.platform
and renovate.organisation = owners.organisation
and renovate.repo = owners.repo
where
package_name = 'github.com/aws/aws-lambda-go'
order by
version desc
$ dmd report gorillaToolkit --db dmd.db
Renovate
Direct dependencies
+------------------------------+---+
| PACKAGE | # |
+------------------------------+---+
| github.com/gorilla/mux | 4 |
| github.com/gorilla/websocket | 1 |
| github.com/gorilla/handlers | 1 |
+------------------------------+---+
Indirect dependencies
+---------------------------------+---+
| PACKAGE | # |
+---------------------------------+---+
| github.com/gorilla/websocket | 6 |
| github.com/gorilla/securecookie | 2 |
| github.com/gorilla/sessions | 1 |
| github.com/gorilla/schema | 1 |
| github.com/gorilla/mux | 1 |
| github.com/gorilla/context | 1 |
+---------------------------------+---+
Working out which Docker namespaces and images you most depend on
$ dmd report mostPopularDockerImages --db dmd.db
Renovate
+---------------------------------------+-----+
| NAMESPACE | # |
+---------------------------------------+-----+
| _ | 346 |
| dockersamples | 12 |
| registry1.dsop.io/ironbank/redhat/ubi | 11 |
| docker.elastic.co/elasticsearch | 6 |
| gcr.io/distroless | 6 |
| cimg | 5 |
| docker.elastic.co/kibana | 4 |
| amazon | 4 |
| gcr.io/kaniko-project | 3 |
| goreleaser | 3 |
| quay.io/something | 3 |
| circleci | 3 |
+---------------------------------------+-----+
+--------------------------------------------+----+
| IMAGE | # |
+--------------------------------------------+----+
| golang | 44 |
| alpine | 36 |
| node | 33 |
| docker | 25 |
| nginx | 21 |
| ubuntu | 19 |
| python | 16 |
| ruby | 15 |
| redis | 14 |
| busybox | 11 |
| registry1.dsop.io/ironbank/redhat/ubi/ubi8 | 11 |
| openjdk | 11 |
+--------------------------------------------+----+
For Go Modules (example query):
select
distinct package_name,
count(*)
from
renovate,
json_each(dep_types) as dep_type
where
package_manager = 'gomod'
and dep_type.value = 'indirect'
group by
package_name
order by
count(*) DESC;
For instance, to get a view of how many updates are pending, per package manager (example):
select
package_manager,
count(*)
from
renovate_updates
group by
package_manager
order by
count(*) desc
Alternatively, how many updates (and whether they're i.e. major bumps) per package manager (example):
select
package_manager,
update_type,
count(*)
from
renovate_updates
group by
package_manager,
update_type
order by
count(*) desc
"we don't like Spring Boot"
select
*
from
renovate
where
datasource = 'maven'
and package_name like 'org.springframework%'
Alternatively:
insert into
advisories (
package_pattern,
package_manager,
version,
version_match_strategy,
advisory_type,
description
)
VALUES
(
'org.springframework*',
'gradle',
NULL,
'ANY',
'OTHER',
'Spring (Boot) is not supported in our organisation. Please see https://internal.docs-site/...'
);
# produce some data that DMD can import, for instance via renovate-graph
npx @jamietanna/renovate-graph@latest --token $GITHUB_TOKEN your-org/repo another-org/repo
# or for GitLab
env RENOVATE_PLATFORM=gitlab npx @jamietanna/renovate-graph@latest --token $GITLAB_TOKEN your-org/repo another-org/nested/repo
# set up the database
dmd db init --db dmd.db
# import renovate-graph data
dmd import renovate --db dmd.db 'out/*.json'
# then you can start querying it
sqlite3 dmd.db 'select count(*) from renovate'