Skip to content

Commit 0164892

Browse files
author
Juliya Smith
committed
Merge pull request #7 in INT/security-event-cli from juliyasmith/INTEG-577/aed-state-mgmt to master
* commit '16f29a4282e3171616bb0c027927bd32914cca6c': (243 commits) Rename common module to util Fix name of function to say value error Try to genericize comment for windows too Fix names of tests to indicate they raise errors not exit Fix error handling tests Use ValueError instead of AttributeError Diversify logger names Use service name in error logger Update comment Update readme Typo Move arg verifications to arg class Test get user proj path Update tests Allow service name into get logger function Allow service name param in get error logger Remove duplicate tests Fill in more tests Move password stuff to common More test fixtures ...
2 parents 498659c + 16f29a4 commit 0164892

19 files changed

+1735
-8
lines changed

‎.gitignore‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,12 @@
1+
# Test config file
2+
*config.cfg
3+
14
# IDE files
25
.idea/
36

7+
# Database files
8+
*.db
9+
410
# Byte-compiled / optimized / DLL files
511
__pycache__/
612
*.py[cod]

‎README.md‎

Lines changed: 181 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,181 @@
1+
# c42seceventcli - AED
2+
3+
The c42seceventcli AED module contains a CLI tool for extracting AED events as well as an optional state manager
4+
for recording timestamps. The state manager records timestamps so that on future runs,
5+
you only extract events you did not previously extract.
6+
7+
## Requirements
8+
9+
- Python 2.7.x or 3.5.0+
10+
- Code42 Server 6.8.x+
11+
12+
## Installation
13+
Until we are able to put `py42` and `c42secevents` on PyPI, you will need to first install them manually.
14+
15+
`py42` is available for download [here](https://confluence.corp.code42.com/pages/viewpage.action?pageId=61767969#py42%E2%80%93Code42PythonSDK-Downloads).
16+
For py42 installation instructions, see its [README](https://stash.corp.code42.com/projects/SH/repos/lib_c42_python_sdk/browse/README.md).
17+
18+
`c42secevents` is available [here](https://confluence.corp.code42.com/display/LS/Security+Event+Extractor+-+Python).
19+
For `c42secevents` installation instructions, see its [README](https://stash.corp.code42.com/projects/INT/repos/security-event-extractor/browse/README.md).
20+
21+
Once you've done that, install `c42seceventcli` using:
22+
23+
```bash
24+
$ python setup.py install
25+
```
26+
27+
## Usage
28+
29+
A simple usage requires you to pass in your Code42 authority URL and username as arguments:
30+
31+
```bash
32+
c42aed -s https://example.authority.com -u security.admin@example.com
33+
```
34+
35+
Another option is to put your Code42 authority URL and username (and other arguments) in a config file.
36+
Use `default.config.cfg` as an example to make your own config file; it has all the supported arguments.
37+
The arguments in `default.config.cfg` mirror the CLI arguments.
38+
39+
```buildoutcfg
40+
[Code42]
41+
c42_authority_url=https://example.authority.com
42+
c42_username=user@code42.com
43+
```
44+
45+
Then, run the script as follows:
46+
47+
```bash
48+
c42aed -c path/to/config
49+
```
50+
51+
To use the state management service, simply provide the `-r` to the command line.
52+
`-r` is particularly useful if you wish to run this script on a recurring job:
53+
54+
```bash
55+
c42aed -s https://example.authority.com -u security.admin@example.com -r
56+
```
57+
58+
If you are using a config file with `-c`, set `record_cursor` to True:
59+
60+
```buildoutcfg
61+
[Code42]
62+
c42_authority_url=https://example.authority.com
63+
c42_username=user@code42.com
64+
record_cursor=True
65+
```
66+
By excluding `-r`, future runs will not know about previous events you got, and
67+
you will get all the events in the given time range (or default time range of 60 days back).
68+
69+
To clear the cursor:
70+
71+
```bash
72+
c42aed -s https://example.authority.com -u security.admin@example.com -r --clear-cursor
73+
```
74+
There are two possible output formats.
75+
76+
* CEF
77+
* JSON
78+
79+
JSON is the default. To use CEF, use `-o CEF`:
80+
81+
```bash
82+
c42aed -s https://example.authority.com -u security.admin@example.com -o CEF
83+
```
84+
85+
Or if you are using a config file with `-c`:
86+
87+
```buildoutcfg
88+
[Code42]
89+
c42_authority_url=https://example.authority.com
90+
c42_username=user@code42.com
91+
output_format=CEF
92+
```
93+
94+
There are three possible destination types to use:
95+
96+
* stdout
97+
* file - writing to a file
98+
* server - transmitting to a server, such as syslog
99+
100+
The program defaults to `stdout`. To use a file, use `--dest-type` and `--dest` this way:
101+
102+
```bash
103+
c42aed -s https://example.authority.com -u security.admin@example.com --dest-type file --dest name-of-file.txt
104+
```
105+
106+
To use a server destination (like syslog):
107+
108+
```bash
109+
c42aed -s https://example.authority.com -u security.admin@example.com --dest-type server --dest https://syslog.example.com
110+
```
111+
112+
Both `destination_type` and `destination` are possible fields in the config file as well.
113+
114+
You can also use CLI arguments with config file arguments, but the program will favor the CLI arguments.
115+
116+
If this is your first time running, you will be prompted for your Code42 password.
117+
118+
If you get a keychain error when running this script, you may have to add a code signature:
119+
120+
```bash
121+
codesign -f -s - $(which python)
122+
```
123+
124+
All errors are sent to an error log file named `c42seceventcli_aed_errors.log`
125+
located in your user directory under `.c42seceventcli/log`.
126+
127+
Full usage:
128+
129+
```
130+
usage: c42aed [-h] [--clear-cursor] [--reset-password] [-c CONFIG_FILE]
131+
[-s C42_AUTHORITY_URL] [-u C42_USERNAME] [-b BEGIN_DATE] [-i]
132+
[-o {CEF,JSON}]
133+
[-t [{SharedViaLink,SharedToDomain,ApplicationRead,CloudStorage,RemovableMedia,IsPublic} [{SharedViaLink,SharedToDomain,ApplicationRead,CloudStorage,RemovableMedia,IsPublic} ...]]]
134+
[-d--debug] [--dest-type {stdout,file,server}]
135+
[--dest DESTINATION] [--dest-port DESTINATION_PORT]
136+
[--dest-protocol {TCP,UDP}] [-e END_DATE | -r]
137+
138+
optional arguments:
139+
-h, --help show this help message and exit
140+
--clear-cursor Resets the stored cursor.
141+
--reset-password Clears stored password and prompts user for password.
142+
-c CONFIG_FILE, --config-file CONFIG_FILE
143+
The path to the config file to use for the rest of the
144+
arguments.
145+
-s C42_AUTHORITY_URL, --server C42_AUTHORITY_URL
146+
The full scheme, url and port of the Code42 server.
147+
-u C42_USERNAME, --username C42_USERNAME
148+
The username of the Code42 API user.
149+
-b BEGIN_DATE, --begin BEGIN_DATE
150+
The beginning of the date range in which to look for
151+
events, in YYYY-MM-DD UTC format OR a number (number
152+
of minutes ago).
153+
-i, --ignore-ssl-errors
154+
Do not validate the SSL certificates of Code42
155+
servers.
156+
-o {CEF,JSON}, --output-format {CEF,JSON}
157+
The format used for outputting events.
158+
-t [{SharedViaLink,SharedToDomain,ApplicationRead,CloudStorage,RemovableMedia,IsPublic} [{SharedViaLink,SharedToDomain,ApplicationRead,CloudStorage,RemovableMedia,IsPublic} ...]], --types [{SharedViaLink,SharedToDomain,ApplicationRead,CloudStorage,RemovableMedia,IsPublic} [{SharedViaLink,SharedToDomain,ApplicationRead,CloudStorage,RemovableMedia,IsPublic} ...]]
159+
To limit extracted events to those with given exposure
160+
types.
161+
-d--debug Turn on debug logging.
162+
--dest-type {stdout,file,server}
163+
The type of destination to send output to.
164+
--dest DESTINATION Either a name of a local file or syslog host address.
165+
Ignored if destination type is 'stdout'.
166+
--dest-port DESTINATION_PORT
167+
Port used when sending logs to server. Ignored if
168+
destination type is not 'server'.
169+
--dest-protocol {TCP,UDP}
170+
Protocol used to send logs to server. Ignored if
171+
destination type is not 'server'.
172+
-e END_DATE, --end END_DATE
173+
The end of the date range in which to look for events,
174+
in YYYY-MM-DD UTC format OR a number (number of
175+
minutes ago).
176+
-r, --record-cursor Only get events that were not previously retrieved.
177+
```
178+
179+
# Known Issues
180+
181+
Only the first 10,000 of each set of events containing the exact same insertion timestamp is reported.

‎aed_config.default.cfg‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
; OPTIONAL CONFIG FILE
2+
; Use this file as an example for which arguments the program accepts.
3+
; Make a copy of this file, edit it, and use with the `-c` flag:
4+
; c42aed -c path/to/config
5+
; Some args may not apply, and you can remove what you do not need.
6+
; You can use this file together with CLI args if you want.
7+
8+
[Code42]
9+
c42_authority_url=https://example.authority.com
10+
c42_username=user@code42.com
11+
begin_date=2019-01-01
12+
end_date=2019-02-02
13+
ignore_ssl_errors=False
14+
output_format=JSON
15+
record_cursor=False
16+
exposure_types=SharedViaLink, SharedToDomain, ApplicationRead, CloudStorage, RemovableMedia, IsPublic
17+
debug_mode=False
18+
destination_type=syslog
19+
destination=https://log.example.com
20+
destination_port=514
21+
destination_protocol=TCP

‎c42seceventcli/aed/args.py‎

Lines changed: 139 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,139 @@
1+
from datetime import datetime, timedelta
2+
from argparse import ArgumentParser
3+
4+
from c42seceventcli.common.cli_args import (
5+
add_clear_cursor_arg,
6+
add_reset_password_arg,
7+
add_config_file_path_arg,
8+
add_authority_host_address_arg,
9+
add_username_arg,
10+
add_begin_date_arg,
11+
add_end_date_arg,
12+
add_ignore_ssl_errors_arg,
13+
add_output_format_arg,
14+
add_record_cursor_arg,
15+
add_exposure_types_arg,
16+
add_debug_arg,
17+
add_destination_type_arg,
18+
add_destination_arg,
19+
add_destination_port_arg,
20+
add_destination_protocol_arg,
21+
)
22+
import c42seceventcli.common.util as common
23+
24+
25+
def get_args():
26+
parser = _get_arg_parser()
27+
cli_args = vars(parser.parse_args())
28+
args = _union_cli_args_with_config_file_args(cli_args)
29+
args.cli_parser = parser
30+
args.verify_authority_arg()
31+
args.verify_username_arg()
32+
args.verify_destination_args()
33+
return args
34+
35+
36+
def _get_arg_parser():
37+
parser = ArgumentParser()
38+
39+
add_clear_cursor_arg(parser)
40+
add_reset_password_arg(parser)
41+
add_config_file_path_arg(parser)
42+
add_authority_host_address_arg(parser)
43+
add_username_arg(parser)
44+
add_begin_date_arg(parser)
45+
add_ignore_ssl_errors_arg(parser)
46+
add_output_format_arg(parser)
47+
add_exposure_types_arg(parser)
48+
add_debug_arg(parser)
49+
add_destination_type_arg(parser)
50+
add_destination_arg(parser)
51+
add_destination_port_arg(parser)
52+
add_destination_protocol_arg(parser)
53+
54+
# Makes sure that you can't give both an end_timestamp and record_cursor
55+
mutually_exclusive_timestamp_group = parser.add_mutually_exclusive_group()
56+
add_end_date_arg(mutually_exclusive_timestamp_group)
57+
add_record_cursor_arg(mutually_exclusive_timestamp_group)
58+
59+
return parser
60+
61+
62+
def _union_cli_args_with_config_file_args(cli_args):
63+
config_args = _get_config_args(cli_args.get("config_file"))
64+
args = AEDArgs()
65+
keys = cli_args.keys()
66+
for key in keys:
67+
args.try_set(key, cli_args.get(key), config_args.get(key))
68+
69+
return args
70+
71+
72+
def _get_config_args(config_file_path):
73+
try:
74+
return common.get_config_args(config_file_path)
75+
except IOError:
76+
print("Path to config file {0} not found".format(config_file_path))
77+
exit(1)
78+
79+
80+
class AEDArgs(common.SecArgs):
81+
cli_parser = None
82+
c42_authority_url = None
83+
c42_username = None
84+
begin_date = None
85+
end_date = None
86+
ignore_ssl_errors = False
87+
output_format = "JSON"
88+
record_cursor = False
89+
exposure_types = None
90+
debug_mode = False
91+
destination_type = "stdout"
92+
destination = None
93+
destination_port = 514
94+
destination_protocol = "TCP"
95+
reset_password = False
96+
clear_cursor = False
97+
98+
def __init__(self):
99+
self.begin_date = AEDArgs._get_default_begin_date()
100+
self.end_date = AEDArgs._get_default_end_date()
101+
102+
@staticmethod
103+
def _get_default_begin_date():
104+
default_begin_date = datetime.now() - timedelta(days=60)
105+
return default_begin_date.strftime("%Y-%m-%d")
106+
107+
@staticmethod
108+
def _get_default_end_date():
109+
default_end_date = datetime.now()
110+
return default_end_date.strftime("%Y-%m-%d")
111+
112+
def verify_authority_arg(self):
113+
if self.c42_authority_url is None:
114+
self._raise_value_error("Code42 authority host address not provided.")
115+
116+
def verify_username_arg(self):
117+
if self.c42_username is None:
118+
self._raise_value_error("Code42 username not provided.")
119+
120+
def verify_destination_args(self):
121+
if self.destination_type == "stdout" and self.destination is not None:
122+
msg = (
123+
"Destination '{0}' not applicable for stdout. "
124+
"Try removing '--dest' arg or change '--dest-type' to 'file' or 'server'."
125+
)
126+
msg = msg.format(self.destination)
127+
self._raise_value_error(msg)
128+
129+
if self.destination_type == "file" and self.destination is None:
130+
msg = "Missing file name. Try: '--dest path/to/file'."
131+
self._raise_value_error(msg)
132+
133+
if self.destination_type == "server" and self.destination is None:
134+
msg = "Missing server URL. Try: '--dest https://syslog.example.com'."
135+
self._raise_value_error(msg)
136+
137+
def _raise_value_error(self, msg):
138+
self.cli_parser.print_usage()
139+
raise ValueError(msg)

‎c42seceventcli/aed/cursor_store.py‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
from c42seceventcli.common.cursor_store import SecurityEventCursorStore
2+
3+
_INSERTION_TIMESTAMP_FIELD_NAME = u"insertionTimestamp"
4+
5+
6+
class AEDCursorStore(SecurityEventCursorStore):
7+
_PRIMARY_KEY = 1
8+
9+
def __init__(self, db_file_path=None):
10+
super(AEDCursorStore, self).__init__("aed_checkpoint", db_file_path)
11+
if self._is_empty():
12+
self._init_table()
13+
14+
def get_stored_insertion_timestamp(self):
15+
rows = self._get(_INSERTION_TIMESTAMP_FIELD_NAME, self._PRIMARY_KEY)
16+
if rows and rows[0]:
17+
return rows[0][0]
18+
19+
def replace_stored_insertion_timestamp(self, new_insertion_timestamp):
20+
self._set(
21+
column_name=_INSERTION_TIMESTAMP_FIELD_NAME,
22+
new_value=new_insertion_timestamp,
23+
primary_key=self._PRIMARY_KEY,
24+
)
25+
26+
def reset(self):
27+
self._drop_table()
28+
self._init_table()
29+
30+
def _init_table(self):
31+
columns = "{0}, {1}".format(self._PRIMARY_KEY_COLUMN_NAME, _INSERTION_TIMESTAMP_FIELD_NAME)
32+
create_table_query = "CREATE TABLE {0} ({1})".format(self._table_name, columns)
33+
insert_query = "INSERT INTO {0} VALUES(?, null)".format(self._table_name)
34+
with self._connection as conn:
35+
conn.execute(create_table_query)
36+
conn.execute(insert_query, (self._PRIMARY_KEY,))

0 commit comments

Comments
 (0)