2020-10-31  Frank Ch. Eigler  <fche@redhat.com>

	* debuginfod.cxx (scan_source_file, scan_archive_file): Add new scanned_bytes_total,

	scanned_files_total metrics.

	(archive_classify): Exit early if interrupted.

	(scan_source_paths): Perform realpath/regex checks only on FTS_F files.

	Tweak metrics.

2020-10-30  Frank Ch. Eigler  <fche@redhat.com>

	PR26775 cont'd.

	* debuginfod.cxx (thread_main_scanner): Ensure control doesn't

	leave infinite loop until program exit, even if SIGUSR2.

	(scan_source_paths): Have traverser clean scanq on

	SIGUSR2.  Emit additional traversed_total metrics.

	(groom): Emit additional groomed_total metrics.

	(thread_main_groom): Restore previous thread_work_total

	metric.

2020-10-29  Frank Ch. Eigler  <fche@redhat.com>

	PR26775

	* debuginfod.cxx (forced_*_count): Make these global.

	(runq::clear): New function.

	(thread_main_scanner): Check for pending SIGUSR2; interrupt.

	(scan_source_paths): Check for pending SIGUSR2; interrupt.

	(groom): Report prometheus stats before groom also.  Check for

	pending SIGUSR1; interrupt.  Increment thread_work_total for

	each file scanned, not the entire cycle.

2020-10-29  Frank Ch. Eigler  <fche@redhat.com>

	PR26810

	* debuginfod.cxx (handle_buildid_*_match): Throw exceptions for

	more lower level libc errors.

	(handle_buildid_match): Catch & report exceptions but return 0

	for continued iteration in the caller.

2020-10-25  Mark Wielaard  <mark@klomp.org>

	* debuginfod-client.c (debuginfod_query_server): Translate

	CURLE_PEER_FAILED_VERIFICATION to ECONNREFUSED.

2020-10-20  Frank Ch. Eigler  <fche@redhat.com>

	PR26756: more prometheus metrics

	* debuginfod.cxx (*_exception): Add counters for error occurrences.

	(fdcache::*): Add counters for fdcache operations and status.

	(fdcache::set_metric): New fn for overall stat counts.

	(fdcache::limit): ... allow metric-less use from dtors.

2020-10-20  Frank Ch. Eigler  <fche@redhat.com>

	* debuginfod.cxx (handle_buildid*): Add a parameter for detecting

	internally-originated lookups for dwz resolution.

2020-09-18  Frank Ch. Eigler <fche@redhat.com>

	* debuginfod.cxx (scan_source_file, archive_classify): Store only

	canonicalized file names in sdef & sref records in the database.

2020-09-08  Mark Wielaard  <mark@klomp.org>

	* Makefile.am (BUILD_STATIC): Include libcurl_LIBS in libdebuginfod

	when NOT DUMMY_LIBDEBUGINFOD.

2020-09-16  Mark Wielaard  <mark@klomp.org>

	* debuginfod-find.c: Fix license block comment.

2020-09-15  Mark Wielaard  <mark@klomp.org>

	* debuginfod-find.c (main): Use dwelf_elf_begin.

2020-07-03  Alice Zhang <alizhang@redhat.com>

	* debuginfod-client.c (debuginfod_query_server): Use strncasecmp

	to compare effective_url. Try CURLINFO_SCHEME as fallback.

2020-06-19  Mark Wielaard  <mark@klomp.org>

	* Makefile.am (bin_PROGRAMS): Guard with DEBUGINFOD and

	LIBDEBUGINFOD.

	(debuginfod_LDADD): Remove libcurl.

	(libdebuginfod): When static and DUMMY_LIBDEBUGINFO remove libcurl.

	(noinst_LIBRARIES): Guard with LIBDEBUGINFOD.

	(AM_CPPFLAGS): Add -Wno-unused-parameter when DUMMY_LIBDEBUGINFOD.

	(pkginclude_headers): Guard with LIBDEBUGINFOD

	(libdebuginfod_so_LIBS): Likewise.

	(+libdebuginfod_so_LDLIBS): Likewise.

	(install): Likewise.

	(uninstall): Likewise.

	* debuginfod-client.c: Include dummy functions when

	DUMMY_LIBDEBUGINFOD.

	* debuginfod.cxx: Remove curl.h include.

2020-06-16  Mark Wielaard  <mark@klomp.org>

	* debuginfod-client.c (debuginfod_query_server): Check malloc.

	Move curl_multi_init call before handle_data malloc call.

2020-06-16  Mark Wielaard  <mark@klomp.org>

	* debuginfod-client.c (debuginfod_query_server): Replace sizeof

	build_id_bytes check with strlen build_id check.

2020-06-16  Mark Wielaard  <mark@klomp.org>

	* debuginfod-client.c (debuginfod_query_server): Increase suffix

	array and prepare having to escape 1 character with 2.

2020-06-16  Mark Wielaard  <mark@klomp.org>

	* debuginfod-client.c (debuginfod_clean_cache): Handle failing

	fopen (interval_path).

2020-03-29  Mark Wielaard  <mark@klomp.org>

	* debuginfod-client.c (debuginfod_add_http_header): Check header

	contains precisely one colon that isn't the first or last char.

2020-03-29  Frank Ch. Eigler  <fche@redhat.com>

	* debuginfod-client.c (struct debuginfod_client): Add a flag field

	for progressfn printing.

	(default_progressfn): Set it if printing \rsomething.

	(debuginfod_end): Terminate with \n if flag set, i.e., only if the

	default_progressfn was actually called.

2020-03-27  Mark Wielaard  <mark@klomp.org>

	* debuginfod.cxx (parse_opt): Check port is not zero.

2020-03-28  Frank Ch. Eigler  <fche@redhat.com>

	* debuginfod.cxx (handle_buildid_r_match): During archive

	extraction / fdcache prefetching, set the mtime of each

	file in the cache.

2020-03-27  Frank Ch. Eigler  <fche@redhat.com>

	* debuginfod-find.c (main): Extract buildid from /binary/ if

	given instead of hex string.

	* Makefile.am: Add elfutils library prereqs for debuginfod-find.

2020-03-24  Frank Ch. Eigler  <fche@redhat.com>

	* debuginfod.h, libdebuginfod.map: New functions for _add_url_header.

	* debuginfod-client.c (struct debuginfod_client): Add headers fields.

	(debuginfod_add_http_header): New client api to add outgoing headers.

	(add_default_headers): Renamed from add_extra_headers, skip if flag.

	(debuginfod_query_server): Pass accumulated headers to libcurl.

	(debuginfod_end): Clean accumulated headers.

	(debuginfod_find_*): Add default headers at this point.

	* debuginfod.cxx (handle_buildid): Add conn pointer.  Use it to relay

	incoming UA and XFF headers to federated upstream debuginfods.

2020-03-26  Frank Ch. Eigler <fche@redhat.com>

	* debuginfod.cxx (handler_cb): Export two families of metrics for

	prometheus traffic analysis: response times and data amounts.

2020-03-26  Frank Ch. Eigler <fche@redhat.com>

	* debuginfod.cxx (parse_opt): For -U, prefer dpkg-deb

	after all if access(3)-able, fallback to bsdtar.

2020-03-25  Frank Ch. Eigler <fche@redhat.com>

	* debuginfod.cxx (parse_opt): Associate a bsdtar subshell with

	the .deb & .ddeb extensions, instead of dpkg-deb.

2020-03-26  Frank Ch. Eigler  <fche@redhat.com>

	* debuginfod-client.c (debuginfod_query_server): Don't

	set CURLOPT_PATH_AS_IS on old curl.  Mostly harmless.

2020-03-24  Frank Ch. Eigler  <fche@redhat.com>

	* debuginfod-client.c (debuginfod_query_server): Set

	CURLOPT_PATH_AS_IS, to propagate file names verbatim.

	* debuginfod.cxx (canon_pathname): Implement RFC3986

	style pathname canonicalization.

	(handle_buildid): Canonicalize incoming webapi source

	paths, accept either one.

	(scan_source_file, archive_classify): Store both

	original and canonicalized dwarf-source file names.

2020-03-24  Frank Ch. Eigler  <fche@redhat.com>

	* debuginfod.cxx (handle_buildid): In case of federated fallback

	queries, handle errors analogously to local ENOENT/404.

	(handle_metrics): Return a size-of-response value.

	(handler_cb): Add code to time entire application-side processing

	stage + response sizes + http codes, so as to emit a complete

	httpd-flavoured log line for each webapi request.

2020-03-24  Frank Ch. Eigler  <fche@redhat.com>

	* debuginfod-client.c (debuginfod_query_server): Print the

	default_progressfn terminating \n message only if that progressfn

	is actually set.

2020-03-24  Frank Ch. Eigler  <fche@redhat.com>

	* debuginfod-find.c (main): Correct /source full-pathness check for

	"debuginfod-find -v source deadbeef /pathname" case.

2020-03-22  Frank Ch. Eigler  <fche@redhat.com>

	* debuginfod-client.c (struct debuginfod_client): Add url field.

	(struct handle_data): Add client field as backpointer.

	(debuginfod_write_callback): Compute & save URL.

	(default_progressfn): Print front pieces of the URL.

	(debuginfod_query_server): Clear URL and cleanup after progressfn.

	* debuginfod-find.c (main): Print URL at transfer conclusion.

2020-03-22  Frank Ch. Eigler  <fche@redhat.com>

	* debuginfod.h, libdebuginfod.map: New functions for _get/set_user().

	* debuginfod-client.c: Implement them.

	* debuginfod-find.c: Include a token call just for testing them.

2020-03-03  Aaron Merey  <amerey@redhat.com>

	* debuginfod-client.c (debuginfod_query_server): Update

	cache_path even when new default path already exists.

2020-02-27  Aaron Merey  <amerey@redhat.com>

	* debuginfod-client.c (xalloc_str): New macro. Call

	asprintf with error checking.

	(debuginfod_query_server): Use XDG_CACHE_HOME as a default

	cache location if it is set. Replace snprintf with xalloc_str.

2020-02-26  Konrad Kleine <kkleine@redhat.com>

	* debuginfod-client.c (debuginfod_query_server): Handle curl's

	response code correctly when DEBUGINFOD_URLS begin with file://

2020-02-25  Frank Ch. Eigler  <fche@redhat.com>

	* debuginfod.cxx (parse_opt): Treat -R as if -Z.rpm .

2020-02-25  Frank Ch. Eigler  <fche@redhat.com>

	* debuginfod.cxx (fdcache_prefetch): New parameter.

	(parse_opt): Parse it.

	(main): Default it.

	(fdcache::fd_size_mb): Change to double for accuracy.

	(fdcache::probe): New function.

	(fdcache::intern): New option to intern at end of LRU.

	(fdcache::lookup): Clean fdcache.

	(handle_buildid_r_match): Implement multi-stage archive

	parsing, with optional prefetching of extracted contents

	into the fdcache.

2020-02-19  Aaron Merey  <amerey@redhat.com>

	* debuginfod-client.c (debuginfod_clean_cache): Restrict

	cleanup to client-pattern files.

2020-02-05  Frank Ch. Eigler  <fche@redhat.com>

	* debuginfod.cxx (argp options): Add -Z option.

	(canonicalized_archive_entry_pathname): New function for

	distro-agnostic file name matching/storage.

2020-01-22  Frank Ch. Eigler  <fche@redhat.com>

	* debuginfod.cxx (dwarf_extract_source_paths): Don't print

	"skipping hat" messages at verbosity <=3, too noisy.

2020-01-19  Frank Ch. Eigler  <fche@redhat.com>

	* debuginfod.cxx (scanq): Rework to let groomer/fts threads

	synchronize with an empty workqueue, and lock out workqueue

	consumers.

	(thread_groom): Adopt new scanq idle APIs to lock out scanners.

	(thread_main_fts_source_paths): Adopt new scanq idler API to

	avoid being restarted while scanners haven't even finished yet.

	(thread_main_*): Increment thread_work_total metric only after

	a work cycle is completed, not when it begins.

2020-01-18  Frank Ch. Eigler  <fche@redhat.com>

	* debuginfod.cxx (thread_main_scanner): Handle empty source_paths[].

2020-01-11  Frank Ch. Eigler  <fche@redhat.com>

	* debuginfod.cxx (libarchive_fdcache): New class/facility to own a

	cache of temporary files that were previously extracted from an

	archive.  If only it could store just unlinked fd's instead of

	filenames.

	(handle_buildid_r_match): Use it to answer dwz/altdebug and webapi

	requests.

	(groom): Clean it.

	(main): Initialize the cache control parameters from heuristics.

	Use a consistent tmpdir for these and tmp files elsewhere.

2020-01-11  Frank Ch. Eigler  <fche@redhat.com>

	* debuginfod.cxx (conninfo): Print User-Agent and X-Forwarded-For

	request headers, after mild safety-censorship (for easier machine

	processing).

2020-01-11  Frank Ch. Eigler  <fche@redhat.com>

	* debuginfod.cxx: Rework threading model.

	(workq): New class for concurrent work-queue.

	(semaphore): Removed class, now unused.

	(scan_source_file_path): Rework into ...

	(scan_source_file): New function.

	(thread_main_scan_source_file_path): Nuke.

	(scan_source_archive_path): Rework into ...

	(scan_archive_file): New function.

	(thread_main_scanner): New function for scanner threads.

	(thread_main_fts_source_paths): New function for traversal thread.

	(scan_source_paths): ... doing this.

	(thread_groom): Tweak metrics for consistency.

	(main): Start 1 traversal and N scanner threads if needed.

2019-01-02  Mark Wielaard  <mark@klomp.org>

	* debuginfod.cxx (default_connect_timeout): Removed.

	(default_transfer_timeout): Removed.

	(default_timeout): New. Default to 90 seconds.

	(debuginfod_query_server): Parse server_timeout_envvar as one number.

	Set as CURLOPT_LOW_SPEED_TIME, with CURL_OPT_LOW_SPEED_LIMITE as 100K.

2020-01-09  Frank Ch. Eigler  <fche@redhat.com>

	* debuginfod-client.c (add_extra_headers): New function,

	based on mjw's draft.

	(debuginfod_query_server): Call it.

2019-12-22  Frank Ch. Eigler  <fche@redhat.com>

	* debuginfod.cxx (*_rpm_*): Rename to *_archive_* throughout.

	(scan_archives): New read-mostly global to identify archive

	file extensions and corresponding extractor commands.

	(parse_opt): Handle new -U flag.

2019-12-19  Frank Ch. Eigler  <fche@redhat.com>

	* debuginfod-client.c (default_progressfn): New function.

	(debuginfod_begin): Use it if $DEBUGINFOD_PROGRESS set.

	(server_timeout): Bump to 30 seconds.

	(debuginfod_query_server): Call progressfn -after- rather than

	before curl ops, to make it likely that a successful transfer

	results in final a=b call.  Tweak cleanup sequence.

	* debuginfod.h: Document $DEBUGINFOD_PROGRESS name.

2019-12-09  Mark Wielaard  <mark@klomp.org>

	* debuginfod-client.c (debuginfod_query_server): Check

	server_urls_envvar early.

2019-12-03  Mark Wielaard  <mark@klomp.org>

	* debuginfod-client.c (debuginfod_query_server): Use separate

	local variables for CURLcode curl_res and CURLMcode curlm_res.

2019-11-26  Mark Wielaard  <mark@klomp.org>

	* Makefile.am (BUILD_STATIC): Add needed libraries for libdw and

	libdebuginfod.

2019-11-25  Frank Ch. Eigler  <fche@redhat.com>

	* debuginfod.cxx (groom): Add a sqlite3_db_release_memory()

	at the end of periodic grooming to try to shrink the process.

2019-11-24  Mark Wielaard  <mark@klomp.org>

	* debuginfod.cxx (test_webapi_sleep): Removed.

	(handler_cb): Don't check test_webapi_sleep and sleep.

	(main): Don't set test_webapi_sleep.

2019-11-24  Mark Wielaard  <mark@klomp.org>

	* debuginfod.cxx (add_metric): New function.

	(scan_source_file_path): Record metrics for

	found_executable_total, found_debuginfo_total and

	found_sourcerefs_total.

	(scan_source_rpm_path): Likewise.

2019-11-07  Frank Ch. Eigler  <fche@redhat.com>

	* debuginfod.cxx: Add /metrics endpoint.  Add numerous

	calls to new functions inc_metric/set_metric to populate

	threadsafe map containing stats.  Add http content-type

	response headers throughout.

	(thread_main_*): Simplify counter/timer flow.

	(main): Reorder web service shutdown to leave http running

	as long as possible.

	* debuginfod.8: Document it, add security caution.

2019-11-06  Frank Ch. Eigler  <fche@redhat.com>

	* debuginfod.cxx: Add new -L (symlink-following) mode.

	* debuginfod.8: Document it.

2019-11-04  Frank Ch. Eigler  <fche@redhat.com>

	* debuginfo-client.c (debuginfod_set_progressfn): New function

	for progress/interrupt callback.

	(debuginfod_clean_cache, debuginfod_query_server): Call it.

	* debuginfo.h: Declare it.

	* debuginfod_set_progressfn.3, *_find_debuginfo.3: Document it.

	* Makefile.am: Install it.

	* libdebuginfod.map: Export it all under ELFUTILS_0.178 symversion.

	* debuginfod-find.c: Add -v option to activate progress cb.

	* debuginfod-find.1: Document it.

	* debuginfod.cxx: Add $DEBUGINFOD_TEST_WEBAPI_SLEEP env var

	to insert sleep in webapi callbacks, to help manual testing.

2019-10-28  Frank Ch. Eigler  <fche@redhat.com>

	* debuginfod.cxx: New file: debuginfod server.

	* debuginfod.8: New file: man page.

	* Makefile.am: Build it.

2019-10-28  Aaron Merey  <amerey@redhat.com>

	* debuginfod-client.c: New file: debuginfod client library.

	* debuginfod.h: New file: header for same.

	* libdebuginfod.map: New file: govern its solib exports.

	* debuginfod-find.c: New file: command line frontend.

	* debuginfod-find.1, debuginfod_find_source.3,

	debuginfod_find_executable.3, debuginfod_find_debuginfo.3:

	New man pages.