Atf Eregs Documentation

Transcript

ATF eRegs Documentation Release 1.0.0 18F Sep 15, 2017 Contents 1 Local (Docker) Setup 1.1 Dev Mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2 Parsing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.3 Other Tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 3 3 4 2 Production Setup 2.1 Architecture . . 2.2 Updating Data . 2.3 Deploying Code 2.4 Services . . . . . 2.5 Non-Cloud.gov . 2.6 Gotchas . . . . . 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 6 7 7 8 8 9 Customization 3.1 Landing Pages . . 3.2 Branded Templates 3.3 Posters . . . . . . 3.4 Style Sheets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 11 12 12 12 4 New Rules 15 5 Documentation Updates 5.1 Build the Docs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 17 i ii ATF eRegs Documentation, Release 1.0.0 These pages explain how to set up ATF eRegulations locally (such as for development), how to set up and deploy it in production, how to customize the ATF-specific content and styles, how to add new rules, and how to update these pages. Contents: Contents 1 ATF eRegs Documentation, Release 1.0.0 2 Contents CHAPTER 1 Local (Docker) Setup We develop using Docker, an open source container engine. If you haven’t already please install Docker and Dockercompose (which is installed automatically with Docker on Windows and OS X). To get started, you will need to clone the ATF-eRegs repository, build the frontend, and run the app: git clone https://github.com/18F/atf-eregs.git cd atf-eregs ./devops/compile_frontend.sh build-dist # Must be run after CSS edits docker-compose up prod Once that’s done, visit the app at http://0.0.0.0:9000/. Dev Mode We also can run in “dev” mode, which includes the Django DEBUG flag and checks out locally-editable versions of regulations-site and regulations-core. Run docker-compose up dev to see this in action. The locally-editable versions of the eregs libraries can be found in the eregs_libs directory, should you need to work on the core code base at the same time as the ATF code base. Note that for the time being, compile_frontend shell script always uses the local version of -site when building scripts. Remember to make a pull request for those upstream changes! Parsing When using the dev and prod applications, we’re pointing to the existing, live data. This is helpful to get running quickly, but if we need to test parsing, we need a local database. To get that set up and kick off parsing, run 3 ATF eRegs Documentation, Release 1.0.0 ./devops/import_data.sh # parses regulations, imports them locally # later, if we just want to see that data docker-compose up dev-with-db And then visit http://0.0.0.0:8001/. The process checks out a copy of regulations-parser in eregs_extensions/eregs_libs/ which can be further edited (similar to -site and -core). Other Tasks Additionally, you can run containerized versions of several Python and Node commands: • docker-compose run --rm manage.py - the Django management command • docker-compose run --rm py.test - our Python test runner • docker-compose run --rm flake8 - a linter for Python • docker-compose run --rm pip-compile - a version-pinning program for Python • docker-compose run --rm grunt - our JavaScript task runner. See regulations-site for more details. Within the eregs_extensions directory, we can similarly run pip-compile, py.test, and flake8 for just the parser extension. 4 Chapter 1. Local (Docker) Setup CHAPTER 2 Production Setup ATF eRegs is currently deployed on cloud.gov, a PaaS which abstracts away many of the annoying components of maintenance. As a result, this documentation (and the production configuration files) are largely geared towards hosting in this environment. See Non-Cloud.gov for other environments. 5 ATF eRegs Documentation, Release 1.0.0 Architecture This repository is a cloud.gov app which stitches together two large Django libraries with cloud.gov datastores and some ATF-specific styles and templates. The first library, regulations-core, defines an API for reading and writing regulation and associated data. atf-eregs mounts this application at the /api endpoint (details about the “write” API will be discussed later). The second library, regulations-site, defines the UI. When rendering templates, regulations-site will first look in atf-eregs to see if the templates have been overridden. These views pull their data from the API; this means that atf-eregs makes HTTP calls to itself to retrieve data (when it’s not already cached). 6 Chapter 2. Production Setup ATF eRegs Documentation, Release 1.0.0 Updating Data When there is new data available (e.g. due to modifications in the parser, new Federal Register notices, etc.), that data must be sent to the /api endpoint before it will be visible to users. However, we don’t want to allow the general public to modify the regulatory data, so we need to authenticate. Currently, this is implemented via HTTP Basic Auth and a very long user name and password (effectively creating an API key). See the HTTP_AUTH_USER and HTTP_AUTH_PASSWORD credentials in cloud.gov for more. Currently, sending data looks something like this (from regulations-parser) eregs pipeline 27 646 https://{HTTP_AUTH_USER}:{HTTP_AUTH_PASSWORD}@{LIVE_OR_DEMO_ ˓→HOSTNAME}/api This updates the data, but does not update the search index and will not clear various caches. It’s generally best to cf restage the application at this point, which clears the caches and rebuilds the search index. Note that this will also pull down the latest versions of the libraries (see the next section); as a result it’s generally best to do a full deploy after updating data. Deploying Code If the code within atf-eregs, regulations-core, or regulations-site has been updated, you will want to deploy the updated code to cloud.gov. At the moment, we build all of the front-end code locally, shipping the 2.2. Updating Data 7 ATF eRegs Documentation, Release 1.0.0 compiled CSS/JS when deploying. This means we’ll need to update our libraries, build the new front end, and push the result. Confusingly, although the front-end compilation step occurs locally, all other library linking (in particular to regulations-site and regulations-core) takes place within cloud.gov. In other words, the setup process for cloud.gov will pull in the latest from regulations-site and regulations-core, regardless of what you have locally and regardless of what you’ve built the front-end against. Be sure to always update your local libraries (via pip) before building and pushing. You’ll need a cloud-foundry shell and the zero-downtime deployment script. To install: curl -L -o cf.tgz "https://cli.run.pivotal.io/stable?release=linux64-binary&version=6. ˓→15.0" tar xzvf cf.tgz -C . ./cf install-plugin autopilot -f -r CF-Community To deploy, log in to cloud.gov (or set the CF_USERNAME and CF_PASSWORD env variables, then: ./devops/deploy.sh dev ./devops/deploy.sh prod # staging # production Services This application uses the aws-rds service on cloud.gov. Services are bound to applications in the manifest files. To create services: cf create-service aws-rds medium-psql atf-eregs-db Our cloud.gov stack should have a user-provided service named atf-eregs-creds including the following credentials: • HTTP_AUTH_USER - at least 32 characters long • HTTP_AUTH_PASSWORD - at least 32 characters long • NEW_RELIC_LICENSE_KEY • NEW_RELIC_APP_NAME To create this service: cf cups atf-eregs-creds -p '{"HTTP_AUTH_USER": "...", "HTTP_AUTH_PASSWORD": "...", ˓→"NEW_RELIC_LICENSE_KEY": "...", "NEW_RELIC_APP_NAME": "..."}' To update, substitute cf uups for cf cups. Non-Cloud.gov If you’d like to move this application off cloud.gov, you will need to create an alternative settings file. Cloud.gov runs the following commands on each deploy (also described in the “manifest” files). Note that these commands are run within the context of a specific settings file, discussed shortly. python manage.py refresh python manage.py collectstatic --noinput gunicorn atf_eregs.wsgi:application 8 Chapter 2. Production Setup ATF eRegs Documentation, Release 1.0.0 The first, refresh, will migrate the database and populate the search index (we’ll see how to configure these in a moment). The command explicitly checks a cloud.gov attribute to determine if it is the “first” server (as we don’t want to run these migrations on each), but it degrades nicely outside of cloud.gov. If not within that environment, running refresh will always run the migrations and rebuild the search index. The second command, collectstatic, gathers all of the static assets (CSS, JS, images, etc.) into one location, for ease of hosting. See the Django docs on that subject for more. Note that this does not build the frontend assets. To do that, you’ll need to run python manage.py compile_frontend first. The final step starts the gunicorn server and points it to a WSGI file. This file sets up a New Relic monitoring agent and wraps the webapp with a static file hosting solution (whitenoise). Django does not host static files, deferring to a variety of other solutions. With a possible exception of the New Relic configuration (which pulls its credentials from cloud.gov), this should all work outside of the cloud.gov environment. Prod Settings The most complicated changes that would need to take place to move off cloud.gov are found in the atf_eregs. settings.prod settings file (which is interwoven as context throughout the deploy script). To make these changes, you may either edit this file (in which case we would strongly recommend committing these changes to a shared repository) or create a local_settings.py file within the same directory and only override the attributes needed. The DATABASES attribute uses the dj_database_url library to pull database connection information from the implicitly defined DATABASE_URL environmental variable. If defining such a variable is not feasible, the DATABASES attribute can be configured manually through traditional settings. The current prod settings file overrides HTTP_AUTH_USER and HTTP_AUTH_PASSWORD with values derived from cloud.gov. By default, however, those credentials are defined by similarly named environment variables. They can also be manually configured in a settings file. These are used to lock down write access to the API. The ALLOWED_HOSTS setting must be configured when DEBUG is turned off. It should contain a list of hostnames that the Django app should respond to; the current prod settings file pulls these from the cloud.gov environment. See Django’s documentation on the topic for more details. Gotchas Caching The production defaults heavily cache eRegs content. As the generated pages change infrequently, most of the content is cached for days. While that works fine for production, it may be rather confusing when initially setting everything up. If you are expecting content to have changed, clear out the eRegs caches: rm -rf /tmp/*eregs* 2.6. Gotchas 9 ATF eRegs Documentation, Release 1.0.0 10 Chapter 2. Production Setup CHAPTER 3 Customization ATF’s eRegs instance is made of several non-agency-specific shared libraries (used by multiple eRegs instances) and a set of ATF-specific customizations. Here we will document these ATF-specific features, deferring to the eRegs docs for broader descriptions of agency-specific customizations (such as a full list of eRegs’ extension points). Landing Pages Each of ATF’s regulations have custom landing page content which is tied to that CFR part. This content is defined in Django’s templating language, with a mix of raw HTML and Django templating tags. For examples, see the five templates of the form atf_eregs/templates/regulations/landing_???.html; each contains blocks of content which get merged into appropriate locations within a larger template. Modifying the content in these files will modify the corresponding landing page. Each of the landing pages derive content from atf_eregs/templates/regulations/ generic_landing.html, which defines the shared legal disclaimer. See Django’s documentation for its templating language for an introduction and comprehensive reference guide. Tags While writing raw HTML into these templates will always work, we strongly recommend using Django’s templating tags to simplify the process of writing content and keeping it up to date. Notably, review the existing landing pages for examples of • url - dynamically link to another part of eRegs • external_link - include aria labeling and visual indicators that this link will open in a new tab • search_for - generate a link to search results 11 ATF eRegs Documentation, Release 1.0.0 Variables Several variables are available for use within your landing page templates. Here we document a few of the more useful. • current_version - a data structure which includes fields corresponding to the unique version identifier for the current (i.e. effective now) version of the regulation • new_version - a data structure including information about upcoming versions of this regulation • reg_first_section - not all regulations start at section 1 • reg_part - the CFR part of this regulation • meta - a data structure containing meta information about this regulation (e.g. its letter, CFR title, etc.) Branded Templates The header and footer of each page have been lightly customized to tie in with ATF’s branding. This includes linking to other ATF sites and including ATF’s logo. If you need to modify the content here, investigate atf_eregs/ templates/regulations/favicon.html, full_footer.html, logo.html, etc. Similarly, about. html defines the content present on the “About” page. Templates in the atf-eregs repository provide customizations that override the core templates provided by the generic/non-agency-specific eRegulations libraries. For example, the about.html file in the atf-eregs repository only contains part of the content you see live on the ATF eRegulations “about” page. This is because most of that “about” page content is provided by the about.html template in the shared/non-agency-specific library (regulations-site). In other words: eRegs supports customized templates primarily through overriding. A base set of templates are provided, broken into components we expect agencies would need to replace. Creating a file of the same name in the atf_eregs/templates/regulations/ folder effectively replaces the template present in the “core” library. This is somewhat risky, as said template might be renamed/moved at a later date (though we strive for backwards compatibility). As such, we recommend only replacing templates on the periphery of functionality (e.g. footers, headers, etc.). For a full list of templates, consult the regulations-site project. Posters The core eRegs application allows the templates used to render specific paragraphs/subparagraphs to be replaced on an ad hoc basis. This mechanism is used to render ATF’s “posters” in 27 CFR 478.103, which contain a header linking to a specific document. Custom templates of this sort must follow a specific naming convention; they must be in the atf_eregs/ templates/regulations/custom_nodes/ directory and must be named according to the paragraph they seek to replace. For example, 478-103-b.html will replace the template normally used to render 27 CFR 478.103(b). Consult the regulations-site documentation for more details. Style Sheets The core eRegs libraries provide a relatively clean set of stylesheets for displaying regulations. To integrate with ATF’s branding, however, these style sheets are modified to use the appropriate colors, fonts, and the like. Similar to the 12 Chapter 3. Customization ATF eRegs Documentation, Release 1.0.0 template overriding process, there exist several SCSS components which can be completely replaced by corresponding files in the atf_eregs/static/regulations/css/scss directory. Consult the regulations-site documentation for full details, but two files are particularly important. • variables.scss defines common variables used throughout other stylesheets. These largely consist of color schemes • module/_custom.less this file exists specifically for ATF-specific modules, such as style sheets related to Posters. After making edits to any of these files, remember to run python manage.py compile_frontend to compile your changes into CSS. 3.4. Style Sheets 13 ATF eRegs Documentation, Release 1.0.0 14 Chapter 3. Customization CHAPTER 4 New Rules eRegs does not have any sort of background process which polls the Federal Register waiting for new data. Further, it does not currently handle eCFR data. Instead, it attempts to derive data from the final rules themselves. Unfortunately, this is a manually triggered, only semi-automated process requiring significant individual review. Until we implement a better solution, however, it works, and allows us to display rules effective in the future. After a final rule has been published in the Federal Register, the parser should be re-ran and the appropriate XML file modified (if needed) to generate a correct set of changes. See the parser’s more thorough docs on this subject to learn more. 15 ATF eRegs Documentation, Release 1.0.0 16 Chapter 4. New Rules CHAPTER 5 Documentation Updates These pages are hosted on Read the Docs, a site that automatically (and for free) provides a formatted display of the documentation files stored in the ATF eRegulations GitHub repository in the docs directory (in the master branch). These files are written in reStructuredText, a markup language. To update these pages: you can edit these files, run the documentation builder (see below), check the generated HTML files to see if they look good, then push the result to the repository. The Read the Docs site will then automatically generate its own HTML files and update these pages. Build the Docs For most edits, you will just need to run the Sphinx documentation builder again. pip install Sphinx cd docs make dirhtml The output will be in docs/_build/dirhtml. If you are adding new modules, you may need to re-run the skeleton build script first: pip install Sphinx sphinx-apidoc -F -o docs regparser/ 17