Merge branch 'issue95' into v2. No conflicts.

With this merge, v2 has all the latest work from master, v1, and issue95.
Devs should base all their new branches off of v2, and existing branches
should be rebased against the tip of v2 so they can be merged back into
v2 when the work is done.  This may be a tricky rebase/merge to do, because
v2 and v1 have diverged widely with the issue95 work.

Author: adobeDan

README.md

@@ -62,7 +62,7 @@ To set up PyCharm for debugging,
 # Basic Usage
 
 ```
-Adobe Enterprise Dashboard User Sync
+User Sync from Adobe
 
 optional arguments:
   -h, --help            show this help message and exit
@@ -127,10 +127,4 @@ optional arguments:
 
 # Configuration
 
-See `examples/example.user-sync-config.yml` for the main configuration template.  The main configuration file user-sync-config.yml must exist in the configuration path.
-
-See `examples/example.dashboard-config.yml` for the dashboard configuration template.  The tool would try and find dashboard-owning-config.yml in the configuration path.
-
-See `examples/example.connector-ldap.yml` for the ldap configuration template.  The main configuration file can be configured to reference this file.
-
-
+See the `examples` directory for sample confguration files of all types.

examples/config files - basic/1 user-sync-config.yml

@@ -0,0 +1,36 @@
+# This is a sample configuration file for the umapi connector type.
+#
+# umapi (user management api) is a network protocol served by Adobe that
+# provides management of users in Adobe-hosted enterprise organizations.
+#
+# This sample file contains all of the settable options for this protocol.
+# All of the settings here can be changed.  It is recommended
+# that you make a copy of this file and edit that to match your configuration.
+# While you are at it, you will likely want to remove a lot of this  commentary,
+# in order to enhance the readability of your file.
+
+# (optional) UMAPI server settings (defaults as shown)
+# These settings specify how to contact the Adobe servers which
+# host the UMAPI services and those which provide authorization.
+# You will never need to alter these settings unless you are provided
+# alternate values as part of a pilot program with Adobe.  It is
+# highly recommended that you leave these values commented out
+# so that the default values are guaranteed to be used.
+server:
+  #host: usermanagement.adobe.io
+  #endpoint: /v2/usermanagement
+  #ims_host: ims-na1.adobelogin.com
+  #ims_endpoint_jwt: /ims/exchange/jwt
+
+# (required) enterprise organization settings
+# You must specify all five of these settings.  Consult the
+# Adobe UMAPI documentation and the Adobe I/O Console to determine
+# the correct settings for your enterprise organization.
+# [NOTE: the priv_key_path setting can be an absolute or relative pathname;
+# if relative, it is interpreted relative to this configuration file.]
+enterprise:
+  org_id: "Org ID goes here"
+  api_key: "API key goes here"
+  client_secret: "Client secret goes here"
+  tech_acct: "Tech account ID goes here"
+  priv_key_path: "path/to/private/key/file"

examples/config files - basic/2 connector-umapi.yml

@@ -1,53 +1,124 @@
-username: "LDAP user goes here"
+# This is a sample configuration file for the ldap connector type.
+#
+# ldap (lightweight directory access protocol) is a network protocol used by
+# most enterprise directory systems (including Active Directory from Microsoft).
+#
+# This sample file contains all of the settable options for this protocol.
+# There is tremendous variation in the user object structure and attribute
+# value structure among LDAP directories even within a single enterprise, so
+# you will likely have to adapt the value specified here to match those in
+# use in your situation.  All of the settings here can be changed, and
+# many do not have default values and so are required.  It is recommended
+# that you make a copy of this file and edit that to match your configuration.
+# While you are at it, you will likely want to remove a lot of this  commentary,
+# in order to enhance the readability of your file.
+
+# connection settings (required)
+# You must specify all four of these settings.  Consult with your
+# enterprise directory administrators to get suitable values.
+# You may want to specify these connection settings in a separate file
+# from the rest of your settings, so as to guard your credential more
+# securely than your other configuration values.  See the User Sync
+# documentation for an explanation of how to do this.
+username: "LDAP username goes here"
 password: "LDAP password goes here"
 host: "LDAP host URL goes here.  e.g. ldap://ldap.example.com"
 base_dn: "defines the base DN. e.g. DC=example,DC=com"
 
-# specifies the string format used to construct a group query.
-# {group} is replaced with the name of the group to find.  Default is:
-# group_filter_format: "(&(|(objectCategory=group)(objectClass=groupOfNames)(objectClass=posixGroup))(cn={group}))"
-#
-# example for AD 
-# group_filter_format: "(&(objectCategory=group)(cn={group}))"
-#
-# example for OpenLDAP
-# group_filter_format: "(&(objectClass=groupOfNames)(objectClass=posixGroup)(cn={group}))"
+# (optional) user_identity_type (default is inherited from main configuration)
+# user_identity_type specifies a default identity type for when directory users
+# are created on the Adobe side (one of adobeID, enterpriseID, federatedID).
+# This overrides the exact same setting in the top-level user sync configuration
+# file, and if not specified here the value set or default there is used as
+# the default value for this connection.  To set an override, uncomment this setting.
+#user_identity_type: enterpriseID
 
-# specifies the string filter used to find all users in the directory.
-# Default, intending for AD, is:
-# all_users_filter: "(&(objectClass=user)(objectCategory=person)(!(userAccountControl:1.2.840.113556.1.4.803:=2)))"
-#
-# example for OpenLDAP
-# all_users_filter: "(&(objectClass=person)(objectClass=top))"
+# (optional) search_page_size (default value given below)
+# search_page_size specifies the result page size requested when
+# fetching values from the directory.
+search_page_size: 200
+
+# (optional) require_tls_cert (default value given below)
+# require_tls_cert forces the ldap connection to use TLS security with cerficate
+# validation.  Allowed values are True (require) or False (don't require).
+require_tls_cert: False
 
-# specifies how an email address is retrieved in the system.
-# the string is a string format, with names enclosed by curly brackets replaced
-# by the corresponding attributes for a user. Default is:
-# user_email_format: "{mail}"
+# (optional) all_users_filter (default value given below)
+# all_users_filter specifies the query used to find all users in the directory.
+# The default value specified here is appropriate for Active Directory, which has a
+# special field that is used to enable and disable users.  The value for OpenLDAP
+# directories might be much simpler: "(&(objectClass=person)(objectClass=top))"
+all_users_filter: "(&(objectClass=user)(objectCategory=person)(!(userAccountControl:1.2.840.113556.1.4.803:=2)))"
+
+# (optional) group_filter_format (default value given below)
+# group_filter_format specifies the format string used to construct a group query,
+# as needed by the --users groups or --users mapped command-line arguments.
+# {group} is replaced with the name of the group to find.  The default value here is
+# complex, because it's meant to work for both AD-style and OpenLDAP-style directories.
+# You will likely want to replace it with a simpler query customized for your directory,
+# such as this one for Active Directory: "(&(objectCategory=group)(cn={group}))"
+# or this one for OpenLDAP: "(&(|(objectClass=groupOfNames)(objectClass=posixGroup))(cn={group}))"
+group_filter_format: "(&(|(objectCategory=group)(objectClass=groupOfNames)(objectClass=posixGroup))(cn={group}))"
+
+# (optional) user_identity_type_format (no default)
+# user_identity_type_format specifies how to construct a user's desired identity
+# type on the Adobe side by combining constant strings with attribute values.
+# Any names in curly braces are take as attribute names, and everything including
+# the braces will be replaced on a per-user basis with the values of the attributes.
+# There is no default value for this setting, because most directories don't contain
+# users with different identity types (so setting the default identity type suffices).
+# If your directory contains users of different identity types, you should define
+# this field to look at the value of an appropriate attribute in your directory.
+# For example, if your directory attribute "idType" had one of the values
+# adobe, enterprise, or federated in it for each user, you could use:
+#user_identity_type_format: "{idType}ID"
+
+# (optional) user_email_format (default value given below)
+# user_email_format specifies how to construct a user's email address by
+# combining constant strings with the values of specific directory attributes.
+# Any names in curly braces are take as attribute names, and everything including
+# the braces will be replaced on a per-user basis with the values of the attributes.
+# The default value used here is simple, and suitable for OpenLDAP systems.  If you
+# are using a non-email-aware AD system, which holds the username separately
+# from the domain name, you may want: "{sAMAccountName}@mydomain.com"
+user_email_format: "{mail}"
+
+# (optional) user_domain_format (no default value)
+# user_domain_format is analogous to user_email_format in syntax, but it
+# is used to discover the domain for a given user.  If not specified, the
+# domain is taken from the domain part of the user's email address.
+#user_domain_format: "{domain}"
+
+# (optional) user_username_format (no default value)
+# user_username_format specifies how to construct a user's username on the
+# Adobe side by combining contstant strings with attribute values.
+# Any names in curly braces are take as attribute names, and everything including
+# the braces will be replaced on a per-user basis with the values of the attributes.
+# This setting should only be used when you are using federatedID and your
+# federation configuration specifies username-based login.  In all other cases,
+# make sure this is not set or returns an empty value, and the user's username
+# will be taken from the user's email.
+# This example supposes that the department and user_id are concatenated to
+# produce a unique username for each user.
+#user_username_format: "{department}_{user_id}"
+
+# Some additional info about LDAP connectors:
+#
+# Unlike the CSV connector, the LDAP connector does not have custom specifications
+# for how to construct user first names, last names, or country codes from the
+# values of different attributes.  That's because the LDAP protocol specifies
+# pre-defined aliases for a large number of typical attribute values, so there
+# are already pre-defined attribute names that are used for these fields:
+# - the Adobe first name is set from the LDAP "givenName" attribute
+# - the Adobe last name is set from the LDAP "sn" (surname) attribute
+# - the Adobe country is set from the LDAP "country" attribute
+# If you need to override these values on the Adobe side, you can use the
+# custom extension mechanism (see the docs) to compute and set field values
+# by combining these and any other custom attributes needed.  Seed the
+# User Sync documentation for full details.
 #
-# other example: 
-# user_email_format: "{sAMAccountName}@example.com"
-
-# specifies the identity type of the dashboard user to create.  
-# the valid values are: enterpriseID, federatedID
-# 
-# If not specified, the default identity type from the main config file is used.
-# 
-# example for enterprise ID:
-# user_identity_type: enterpriseID
-
-# specifies the result page size.  Default is:
-# search_page_size: 200
-
-# set to True if you want to validate SSL cert.  Default is:
-# require_tls_cert: False
-
-# Definition of where in the directory to get the domain if that information 
-# is in a non-standard place.  The value can be a fixed string and/or one or
-# more directory attribute names enclosed in curly braces.
-# user_domain_format: {domain}
-
-# Definition of where in the directory to get the user name for a federated
-# domain using username-based login.  The value can be a string and/or one or
-# more directory attribute names enclosed in curly braces.  For example,
-# user_username_format: {user_id}_{department}
+# Finally, some LDAP systems use uids to identify groups, and place users in
+# groups via uid rather than name.  The User Sync implementation always reads
+# the uid attribute on all objects if the directory provides one, so it is
+# able to handle directories which function in this way even though the
+# configuration files always specify groups by name.

examples/config files - basic/2 dashboard-config.yml

@@ -0,0 +1,73 @@
+# This is a sample configuration file for the csv connector type.
+#
+# CSV (Comma-Separated Values) is a plain-text spreadsheet format.
+# The main function of this configuration file is to specify how column names
+# in the spreadsheet are mapped to attribute names in the customer directory.
+#
+# The first line of a CSV file is assumed to be header row with column names.
+# The column names in your spreadsheet can be in any order, and if there are
+# fewer values in a row than there are in the header row, the attributes
+# for the missing columns are given no value.
+#
+# This sample file contains all of the settable options for this format,
+# with each set to its default value.  If the defaults are fine for your
+# application, you can use a copy of this file as-is, or you can omit the csv
+# setting from the connectors in your main User Sync configuration file.
+
+# (optional) delimiter (no default value)
+# The delimiter is the string that separates columns.  Normally,
+# this is detected automatically from the first line in the file.
+# To set it to a specific value, uncomment this setting:
+#delimiter: ","
+
+# (optional) email_column_name (default "email")
+# The column name that contains the user's email address.
+# Values in this column must be valid, unquoted email addresses.
+# A value is required in this column for all users, regardless
+# of their identity type.  For Adobe ID users, all of the other
+# column values are ignored.  For Enterprise and Federated ID
+# users, all of the other column values are significant.
+email_column_name: email
+
+# (optional) first_name_column_name (default "firstname")
+# The column name that contains the user's first name (aka given name).
+# Values in this column can be any string
+first_name_column_name: firstname
+
+# (optional) last_name_column_name (default "lastname")
+# The column name that contains the user's last name (aka surname).
+# Values in this column can be any string
+last_name_column_name: lastname
+
+# (optional) country_column_name (default "country")
+# The column name that contains the user's home country.
+# Values in this column must be an ISO-3166 two-letter country code.
+country_column_name: country
+
+# (optional) groups_column_name (default "groups")
+# The column name that contains the user's group memberships.
+# Values in this column must be a comma-separated list of group names.
+# NOTE: Since commas usually separate columns, be sure to surround the
+# entire column value with double quotes, to prevent any embedded commas
+# from being interpreted as column separators.
+groups_column_name: groups
+
+# (optional) identity_type_column_name (default "type")
+# The column name that contains the user's identity type.
+# Values in this column must be adobeID, enterpriseID, or FederatedID.
+identity_type_column_name: type
+
+# (optional) username_column_name (default "username")
+# The column name that contains the user's username (for the Adobe side).
+# Values in this column should not be specified unless the user is of type federatedID
+# and the user's domain is configured for username-based federation.  In all other
+# cases, leave this column blank, and the email will be used for the username.
+username_column_name: username
+
+# (optional) domain_column_name (default "domain")
+# The column name that contains the user's domain.
+# Values in this column should not be specified unless the user is of type federatedID,
+# the user's domain is configured for username-based federation, and there is a value
+# in the username field.  If this field is left blank, the domain part of the email
+# address will be used for the user's domain.
+domain_column_name: domain

examples/config files - basic/3 connector-ldap.yml

@@ -0,0 +1,61 @@
+# This is a sample configuration file for a user extension.
+#
+# This file extends the behavior specified in your main configuration file,
+# by adding custom python code to the logic for attribute and group mapping
+# which is specified in the directory_users section.
+#
+# To load your extension file, put the file's relative or absolute path
+# as the value of the extension setting in the directory_users section
+# of your main configuration file.
+
+# (optional) extended_attributes (default value is an empty list)
+# extended_attributes is a list of attribute names whose per-user
+# values are required by your extension in order to function properly.
+# These attributes will be read on a per-user basis, and will be available
+# in the source_attributes dictionary in your after_mapping_hook.  Any
+# of these attributes which don't have a value in the directory entry for
+# a given user will have a Python None value in that user's dictionary.
+extended_attributes:
+  - bc
+  - subco
+
+# (optional) extended_adobe_groups (default value is an empty list)
+# extended_adobe_groups is a list of Adobe-side product configuration
+# and/or user group names, exactly like those found in the groups
+# setting in the main configuration file.  Your after_mapping_hook
+# can add users to any product configuration or user group found here
+# as well as any found in the groups setting, and the effect of the
+# --process-groups argument will treat them exactly as if the
+# extended mapping had been specified as part of the groups setting.
+extended_adobe_groups:
+  - Company 1 Users
+  - Company 2 Users
+
+# (required) after_mapping_hook
+# This is where you specify your Python hook code.  Note the vertical bar
+# after the after_mapping_hook label: this vertical bar is required and
+# denotes that all the following indented lines up to the next blank
+# line are part of a code block.  Do not have blank lines in your code block.
+#
+# after_mapping_hook code executes in a scope containing the following variables:
+#
+#     source_attributes   # in: attributes retrieved from customer directory system (eg 'c', 'givenName')
+#                         # out: N/A
+#     source_groups       # in: customer-side directory groups found for user
+#                         # out: N/A
+#     target_attributes   # in: user's attributes for UMAPI calls as defined by usual rules (eg 'country', 'firstname')
+#                         # out: user's attributes for UMAPI calls as potentially changed by hook code
+#     target_groups       # in: Adobe-side dashboard groups mapped for user by usual rules
+#                         # out: Adobe-side dashboard groups as potentially changed by hook code
+#     hook_storage        # for exclusive use by hook code: initialized to None; persists across per-user calls
+#     logger              # an object of type logging.logger which outputs to the console and/or file log
+#
+after_mapping_hook: |
+  bc = source_attributes.get('bc')
+  subco = source_attributes.get('subco')
+  if bc is not None:
+    target_attributes['country'] = bc[0:2]
+  if subco == 'Company 1':
+    target_groups.add('Company 1 Users')
+  elif subco == 'Company 2':
+    target_groups.add('Company 2 Users')